Senior Data Engineer - Spark/Kafka (3-9 yrs) Bangalore (Backend Developer)
Colosseum Consulting
Bangalore
6d ago
source : hirist.com

Responsibilities :

  • Build tools for the data pipeline process. Orchestrating workflows, handling metadata and transformations- there will be a special emphasis on having good interfaces to those tools so that evolving them with time and integrating other data management tools into existing structures will be easier.
  • Play a key role in ensuring best practices for deployment & metrics, and keep track of those processes.
  • Leverage tools available in the open-source community and take active steps in giving back to the community we love.
  • Write code and documentation, participate in code reviews and collaborate with other engineers. (Every engineer who joins us must believe in the importance of communication.)
  • Skillsets required :

  • You have relevant industry experience of 4+ years working with large scale distributed systems
  • You have experience building and optimizing data pipeline architectures.
  • You have strong knowledge of concepts such as batch and stream processing, distributed querying, message queues and data storage techniques.
  • You are familiar with tools like Presto, Spark, Dask, Kafka and Airflow.
  • You have experience with computing services from any one major cloud provider like AWS, Azure etc.
  • You are familiar with containerization technology and tools such as Kubernetes.
  • You have experience working with and contributing to open-source projects (being a strong member of the community is a plus).
  • Report this job
    checkmark

    Thank you for reporting this job!

    Your feedback will help us improve the quality of our services.

    Apply
    My Email
    By clicking on "Continue", I give neuvoo consent to process my data and to send me email alerts, as detailed in neuvoo's Privacy Policy . I may withdraw my consent or unsubscribe at any time.
    Continue
    Application form