Working with a modern analytics stack, including Google Cloud, BigQuery, Informatica and Python, the Lead Data Engineer will be building data pipelines, transforming, structuring, and operationalising data to support BI, Analytics and Data Science projects.
responsibilities :
Contribute to the development and design of the data platform, including the design and execution of data extraction, transformation, and optimal data pipeline architectures and solutions at an enterprise scale to enable business intelligence, analytics, and data science use-cases
Design and implement processes around data-loading, data-transformation and feature engineering as well as access control, data security and data privacy
Develop complex code-based ETL/ELT data pipelines with performance optimised data modelling
Develop source of truth data models to power business analytics solutions using data warehouse modelling techniques, data quality checks, reconciliation processes and thorough testing practices
Establish and implement best practices in coding, monitoring and alerting, using CI/CD and other relevant techniques
Ensure code is manage, validated, and released using best practices (DevOps) making effective use of automation for these processes
Maintain a repeatable and automated approach to data management including a continuous deployment pipeline, centralised configuration management, automated testing patterns, provisioning of environments and controlled release to production
Ensure all features and new releases are tested, upgraded, and changed as necessary from a data perspective
Take proactive action to ensure that WHS practices and policies are in place to provide a safe environment for themselves and all team members, and adhere to legislative requirements, company expectations and other safety initiatives
Follow safe work and operating procedures at all times
Ensure the work area is kept in a neat and tidy manner, free from hazards
Promptly report all OHS hazards, injuries and incidents
Understand and demonstrate behaviour in accordance with all Company policies and procedures, including Code of Conduct, Discrimination, Bullying and EEO and all applicable laws whilst ensuring continual compliance with relevant legislation relevant to the role including WHS
All other tasks as directed by Manager
requirements-expected :
Proficiency in designing efficient and robust ETL/ELT workflows, data modelling and data pipelines – batch and/or event driven
Proficiency in designing data solutions and creating solution architecture
Experience working in a complex enterprise data warehouse environment and developing and maintaining pipeline processes to move and transform data
Experience with data warehousing principles, including orchestration framework, schema structure and data architecture principles
Extensive experience with Google Cloud Platform (GCP)
Experience with Informatica ETL and Governance
Ability to write robust code in SQL, and Python
benefits :
sharing the costs of sports activities
private medical care
sharing the costs of professional training & courses
