We are seeking a talented Big Data Engineers to work on Qualys’ next-generation big data challenge. Working with a team of engineers and architects, you will be responsible for the creation of data lakes by extracting and transforming data from various data sources.
As a Big Data Engineering team member, you would be responsible for the development, implementation, and support of critical data intelligence ETL solutions.
This is a great opportunity to be an integral part of Qualys’ next-generation data technology platform by leveraging open source technologies and work on challenging and business-
impacting projects. Develop, deploy and maintain Big Data stack applications and data services, work on collecting, storing, processing, validating and analyzing huge sets of data.
Develop and support data flows, ETLs, UDFs and machine learning processes using Java, Scala, Python, R within Hadoop / Spark / Flink / Hive / Presto / Drill programming API.
Work with application developers and analytic teams to build data marts and snapshot tables for Data Warehousing and Data Lake management.
Monitor data transformations performance and define data retention policies. Develop scalable and high-performance data services for high-
speed querying. Maintain security and data privacy in the distributed environment. Test data processing prototypes and oversee handover to operational teams.
Must have experience with Big Data querying tools, such as Hive, Presto, Drill, Impala, BigQuery, Athena and strong knowledge of SQL (including DDL) and relational data structures.
Must have experience with developing of Spark, Flink applications and / or Hadoop MapReduce batch processing jobs. Experience with NoSQL databases, such as Cassandra and NoSQL data structures.
Knowledge of various ETL techniques and frameworks. Experience with various messaging systems, such as Kafka or RabbitMQ.
Ability to develop code in Java (Scala, Python, and R are all beneficial including experiences with Numpy / Pandas / Sci-
kit). Ability to leverage and partner business intelligence tools such as Power BI, Tableau, MicroStrategy.