Data Engineer (Python Developer)

Your responsibilities

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.
• Development of CI/CD pipelines (esp. GitLab and DataOps)
• Modeling data transformations using dbt (data build tool), including scripting and complex algorithms
• Automation and simplification of processes (Bash, Docker, Python, GitLab, JIRA)
• Proficient use of AWS services, including but not limited to S3, Step Functions, Lambda, Fargate, AWS Glue, RDS. Strong hands-on experience with ETL (Extract, Transform, Load) processes, including the design and implementation of ETL solutions using AWS technologies.
• Strong experience in Python development, including writing clean, efficient, and maintainable code for data processing and automation tasks.

Our requirements

• Python (for data processing and automation tasks)
• AWS Services (S3, Step Functions, Lambda, Fargate, Glue, RDS)
• SQL (for data extraction, transformation, and loading)
• CI/CD & DataOps (experience with GitLab and building automation pipelines)