We are seeking highly skilled and experienced DataOps Engineer who will join a team working in a global container liner shipping company. Team is specialising in building Data Lake and leveraging the Databricks platform. This role is an exciting opportunity to lead the development and optimization of data orchestration platform, leveraging Astronomer and Airflow, and enabling CI/CD workflows for a cutting-edge, cloud-based Data Warehouse built on Databricks and AWS.
Possibility of 100% remote work.
responsibilities :
Developing and maintaining a scalable Astronomer-based scheduling environment for our Data Lakehouse platform
Designing and implementing CI/CD pipelines to automate deployments, validate data models, and monitor workflows
Configuring and managing Astronomer workspaces, deployments, and access control
Integrating dbt Core with Cosmos for orchestrating dbt models in Airflow
Building and optimizing Cosmos Airflow DAGs for data ingestion and transformation
Documenting and implementing deployment processes, including ephemeral environments for development
Collaborating with stakeholders to ensure best practices in pipeline design and Airflow job scheduling
Leveraging AWS services (e.g., S3) and Databricks for performance, scalability, and security
Troubleshooting and supporting data scheduling, CI/CD processes, and model deployment pipelines
Ensuring comprehensive documentation of workflows, CI/CD processes, and operational standards
Finalizing the migration of the existing Data Lake scheduling from Airflow MWAA to Astronomer
requirements-expected :
At least 5+ years of experience as a Data Engineer or DataOps Engineer, focusing on CI/CD pipelines and workflow orchestration
Expertise in configuring and deploying Astronomer or Apache Airflow in cloud environments
Hands-on experience with Databricks, dbt Core, and AWS services
Proficiency in Python and Spark for data transformation and pipeline development
Strong SQL skills and knowledge of relational databases
Familiarity with GitFlow and CI/CD tools like GitLab or Jenkins
Knowledge of dimensional modeling, data access patterns, and data storage optimization
Experience with Terraform for Infrastructure as Code and Airflow DAGs with Cosmos
Excellent collaboration and communication skills, with a proactive approach to sharing knowledge
Proficiency in English at the C1 level
offered :
Financial transparency: the salary is: 26 880 – 31 920 PLN (160 – 190 PLN/h) + VAT
Clear wage model. 7N Consultants know exactly what’s 7N fee for its agent role on the project
Professional development support. We subsidize trainings, technical certificates, conference participation, and foreign language classes. Additionally, 7N Consultants have opportunities to take part in internal training sessions and develop their interpersonal skills within the framework of the 7N Secret Code
Access to e-learning platforms: internal – 7N Academy with recordings of technical trainings (e.g. AWS, Azure, AI, DevOps, Data Science, etc.), inspiring webinars (e.g. healthy eating, personal finance, language learning, etc.), and external platforms: Coursea, Mentorix
Cloud Path - a development program that includes online trainings on AWS and Azure, ending in a certification exam. Each participant receives access to e-learning platforms: Microsoft ESI, Google Cloud Platform
An opportunity to join 7N Inspiration Team - a group of 7N Consultants who consistently and actively share their professional knowledge and expertise
Career Development Program – a mentoring program that includes exploring your individual predispositions and preferences for career paths; providing insight into the latest trends and valuable positions in the IT industry. Participants meet with Career Agents and/or 7N Mentors
Cooperation with real experts – the average professional experience of a 7N consultant is 10 years
Comprehensive and personal project support from one of a 7N Agent. Priority regarding project continuity and quality
Numerous (10-12 each year) high-end events – both online and offline
Healthcare, Benefit Multisport and life insurance subsidies regardless of cooperation model. Access to Mindgram, a psychological support and professional development platform
Professional and quality-oriented recruitment process led by the most experienced recruiters in the industry
benefits :
sharing the costs of sports activities
private medical care
sharing the costs of foreign language classes
sharing the costs of professional training & courses
