Job Purpose :
A data engineer works on implementing complex data projects with a focus on collecting, parsing, managing, analysing and visualising large sets of data to turn information into value using multiple platforms.
You will work with business analysts and data scientists to understand customer business problems and needs, secure the data supply chain, implement analysis solutions and visualise outcomes that support improved decision making for a customer.
You will understand how to apply technologies to solve big data problems and to develop innovative big data solutions.
As the pace of technological advancement continues to quicken, the way we are operating and serving our customers is developing further.
Huge increase in the volume and sources of data presents massive opportunities to reveal new insights that have previously remained hidden, creating new avenues to discover untapped value within the Group, and for our customers.
Key Accountabilities :
Working with colleagues to understand and implement requirements
Securing the data supply chain, understanding how data is ingested from different sources and combined / transformed into a single data set.
Understanding how to analyse, cleanse, join and transform data.
Implementing designed / specified solutions into the chosen platform (e.g. Azure Data Factories / Data Lakes, HDInsight, Talend, MuleSoft or traditional software).
Working with colleagues to ensure that the on-prem / cloud infrastructure available is capable of meeting the solution requirements.
Planning, designing and conducting tests of the implementations, correcting errors and re-testing to achieve an acceptable result.
Appreciate how to manage the data including; security, archiving, structure and storage.
Key Experience and Qualifications :
Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience,
10-15 years of experience at various levels of Software / Data Engineering roles
8+ years of experience in designing solutions using databases and data storage technology such as RDBMS, NoSQL, MongoDB, Hadoop, Cassandra
Be up to date with data processing technology / platforms such as Spark, PowerBI, and Tableau.
Experience with ETL and / or data integration tools such as Informatica, SSIS, Talend, MuleSoft, Dell Boomi
Experience of working in at least one of the public cloud platform such as Azure, AWS, Google Cloud and using IaaS / PaaS / SaaS components in building highly scalable solutions
Good understanding of infrastructure components and their fit in different types of data solutions
Exposure to scripting and automation such as PowerShell, R, Python
Experience of designing solutions deployed on Microsoft and Linux operating systems
Exposure to DevOps
Experience of working in an agile environment, within a self-organising team.
Experience with Microsoft development stack (C#.net)
Experience with time series data processing and storage including technologies such as InfluxDB, OpenTSDB etc
ref : hirist.com)