Data Engineer I Big Data Ecosystems | Dirty Mirrors | 3-6 years
Dirty Mirrors
Indiranagar, Bengaluru, Karnataka, India
3d ago

Key goals for the position :

  • Develop technical solutions for a data platform based on existing and emerging patterns and technologies
  • Enable advanced analytics including AI / ML and data-driven decision making by building pipelines that are event driven, can transform / cleanse / manipulate data and facilitate visualization of data integrated from enterprise systems, internet, and many other disparate sources
  • Enable data exchange amongst enterprise systems via real time event streaming platform
  • Champion conceptual and technical aspects of real time data platform development and support, using various design patterns, dynamically scalable technologies, and Devops practices
  • Team Member Minimum Requirements

    Preferred Formal Education / Qualifications :

  • Bachelor of Computer Science
  • Other qualifications in IT or related discipline Proven Experience :
  • Developing, Testing and Supporting data lakes, data warehouses and large-scale data processing systems.
  • Working effectively in data platform scrum teams for technical design, development and deployment of solutions.
  • Implementing and supporting data platform technologies both on-premise and cloud-based.
  • Deep understanding and adoption of Agile delivery techniques, including Continuous Integration & Continuous Delivery (CI / CD).
  • Developing solutions using a variety of technologies and tools to marry on-premise and cloud-based systems together.
  • Participate in technology / tools / framework evaluation to recommend / influence adoption
  • Follow best practices to ensure high standards of data availability, reliability, completeness, efficiency and quality.
  • Skills / Knowledge / Abilities / Technology Used :

  • Advanced SQL working knowledge and experience working with a variety of relational databases, SQL query authoring.
  • Expert understanding and hands-on working knowledge of message queueing, real time event streaming architecture for data platforms including Kafka, Kinesis, SNS, SQS etc.
  • Deep understanding and ability to build automations for data transformation, processing of data structures, metadata, dependencies and workload management for very high volume, velocity and variety of data.
  • Deep working knowledge of AWS technologies such as S3, EC2, EMR or Bigdata / Hadoop, RDS, Lambda, Elasticsearch, Redshift, Cloudformation, Terraform, Cloudwatch etc.
  • Sound to advanced level working knowledge of Git and CI / CD pipeline technologies such as Jenkins, Chef, Kubernetes & Docker containers
  • Deep experience with object-oriented / object function scripting languages : Python, Scala, Java, R, C++, Golang etc.
  • Working knowledge of Data warehouses, NoSQL databases and ETL technologies like Informatica, Talend etc
  • Understanding of modern SaaS based monitoring and logging tools like New Relic / Sumologic or equivalent
  • Must have experience with Linux, shell scripting
  • Familiarity with BI tools a plus
  • Area of Accountability Key Responsibilities & Deliverables Performance Measures & Targets

    Development and Operations :

  • Collaborate with team members to design and implement data solutions in alignment with the project schedule.
  • Code, test, and document new or modified data systems to create robust and scalable applications for data analytics.
  • Peer and customer feedback
  • Line manager observations
  • Line manager observations
  • Create data flow diagrams for business systems.
  • Implement security as part and parcel of all development.
  • Builds automation tools to provide a self-service data platform and enable CICD
  • Creates and maintains a data catalogue.
  • Develops standards and processes.
  • Champions change management
  • Take Care Safety
  • Accountable for a safe site for everyone, every day by implementing and evaluating safe work practices, improving safety performance and
  • celebrating safety achievements

  • Follows a Devops mindset and practice
  • Communication :

  • Conducts product demonstrations, showcases, briefings on trending technologies, processes and solutions relating to the data platform.
  • Line manager observations
  • Line manager observations
  • Communicates effectively with stakeholders, partners, vendors.
  • Peer and customer feedback
  • Other Areas of Accountability Key Responsibilities / Major Activities :

    Values and Behaviour

  • Live Integrity
  • Think Customer
  • Grow Together
  • Reach Higher
  • Report this job

    Thank you for reporting this job!

    Your feedback will help us improve the quality of our services.

    My Email
    By clicking on "Continue", I give neuvoo consent to process my data and to send me email alerts, as detailed in neuvoo's Privacy Policy . I may withdraw my consent or unsubscribe at any time.
    Application form