Design and build reliable data pipelines to efficiently ingest, combine, and transform data from multiple sources, including internal systems, IoT, fleet data, and external providers
Guarantee smooth and consistent integration of data into the Data Lake while meeting data quality and integrity standards
Apply advanced knowledge of Databricks and Apache Spark to develop, enhance, and optimise data processing workflows within the Data Lake
Use Spark to handle large-scale data processing, execute complex transformations, and support data aggregation for analytical purposes
Take responsibility for data storage architecture and management in AWS, including organising data across appropriate buckets and zones
Partner with IT administrators to maintain proper access management and ensure strong data security controls
Implement data tokenisation solutions to protect sensitive information and comply with data protection regulations
Regularly monitor Data Lake performance and proactively identify areas for optimisation
Improve pipeline efficiency and storage design to increase data access speed and overall system performance
Work closely with the Product Group (data owners) to understand business needs, offer technical expertise, and maintain data quality across the entire data lifecycle
Collaborate with data scientists to ensure data accessibility and readiness for analytical and business use cases
Provide advanced technical support to Data Lake users and the engineering team, resolving issues related to ingestion, processing, and access
Investigate, troubleshoot, and fix data-related incidents promptly and effectively
Advocate and enforce best practices in data governance, ensuring compliance with data standards and proper documentation
Keep thorough and up-to-date documentation of data pipelines, processes, and data lineage for transparency and future reference
Wymagania
Minimum 5 years of professional experience as a Data Engineer, with strong emphasis on designing and maintaining Data Lake architectures
Extensive knowledge of AWS services, particularly in data storage and processing using S3, Databricks, and Spark
Proven hands-on experience with Databricks, Apache Spark, and Python for large-scale data processing and performance optimisation
Strong proficiency in big data file formats such as Parquet, Iceberg, and Delta
Good understanding of data protection regulations and best practices for handling personally identifiable information (PII)
Familiarity with data visualisation and reporting tools (e.g., Qlik Sense) is an advantage
Experience with containerisation technologies and Infrastructure as Code, particularly Terraform
Practical knowledge of workflow orchestration tools such as Airflow
Proficiency in English
Oferujemy
Ongoing support from a dedicated agent, taking care of your project continuity, client contact, necessary formalities, work comfort and development
Consultant Development Program – advice on growth planning based on the latest trends and market needs in IT, including consultations with agents and growth mentors
Access to 7N Learning & Development – a development and educational platform with webinars, a library of articles and industry reports, and regular invitations to one-time and recurring development events – technical, business, and lifestyle
Professional development not only during the project – you can get involved in knowledge transfer to others within the 7N Services offering directed at 7N clients
Relationships and access to the knowledge of the most experienced IT experts in the market – the average professional tenure of our consultants in Poland is over 10 years
A complete benefits package, including funding for medical care, life insurance, sports cards for you and your loved ones, as well as discounts in stores in Poland and abroad
