The Data Engineer is a software engineer specialised in data solutions, including technologies to use and scalability of different solutions.
Typically reports to the Chapter Lead.
Designs, constructs, installs, tests and maintains highly scalable data management systems, to business requirements and industry practices through the application of data modelling, data warehousing and data manipulation techniques and standards;
Builds and deploys individual datasets for further modeling based on commercial and / or analytical requirements by extracting, loading and transforming data from different sources;
Supports integration of new data sources ( batch and streaming) into shared data environment ( EDW, Big Data Platform, etc.);
Builds high-performance algorithms, prototypes, predictive models and proof of concepts, translating business requirements into concrete data engineering tasks;
Researches opportunities for data acquisition and new uses for existing data;
Develops data set processes for data modelling, mining and production;
Integrates new data management technologies and software engineering tools into existing structures;
Creates custom software components and analytics applications;
Employs a variety of languages and tools ( scripting languages) to marry systems together;
Installs and updates disaster recovery procedures;
Recommends ways to improve data reliability, efficiency and quality;
Collaborates with data architects, modellers and IT team members on project goals;
Designs, develops and maintains programmes ( written in Python, Scala, Java);
Designs, creates and maintains reports and dashboards.
Data ModellingData Analytics and InsightsData Management and TransformationImplementation and IntegrationCodingSoftware EngineeringAutomation and Robotics 2022Testing and EvaluationComplexity ManagementData Visualisation