designing and implementing extremely efficient data ingestion and wrangling pipelines from various kind of sources (volumes up to hundreds of terabytes) into Delta Lake (Azure Databricks);
orchestrating data processing on various stages;
creating and executing automated tests for the purposes of validation;
providing insights on the latest technologies which might be later used by clients
Must-have:
3+ years of experience in area of data processing, esp. Databricks, including optimizations;
familiar with Databricks, Delta Lake and its core concepts;
good knowledge of Microsoft Azure cloud services (Azure Data Factory, Azure Key Vault, Azure Data Lake Storage Gen2);
programming skills in Python;
proficiency in Spark/Databricks SQL;
experience in collaborative work on code using Git.
Nice-to-have:
familiar with DataOps approach;
cloud certificates;
good knowledge of English (spoken and written).
We offer:
Work in a team of interdisciplinary experts and technology enthusiasts, inspired by working in a flat hierarchy;
Permanent full-time cooperation in an extraordinary company, which has been growing dynamically for more than 20 years;
Possibility of development in various IT projects realized for well-known and recognizable clients – leaders in their industries;
The ability to work with the latest technologies and the highest standards of development;
⌚ Flexible working hours and a hybrid or fully remote working model (different arrangements apply in different teams, we do not impose any options);
☁ A number of initiatives supporting knowledge sharing and development, such as: internal training, thematic guilds, Cloud training and certification opportunities (technology partnerships in four clouds);
Possibility to learn English or German online and English lessons for teams;
♂ Multisport card and medical insurance;
⛵ Sea adventures co-financed by company – we regularly sail in the Caribbean, Mediterranean or North Sea.
