We are seeking two highly skilled and experienced Data Engineers who will join a team working in a global container liner shipping company.
Team is specialising in building Data Lake and leveraging the Databricks platform.
Possibility of 100% remote work.
responsibilities :
Designing and implementing robust data processing pipelines to efficiently ingest, integrate, and transform raw data from diverse sources such as internal databases, IoT data, fleet data, and third-party sources
Ensuring seamless integration of data into the Data Lake, adhering to data quality and consistency standards
Utilizing your strong expertise in Databricks and Apache Spark to develop and optimize data processing workflows within the Data Lake
Leveraging Sparks capabilities to process data at scale, perform complex transformations, and facilitate data aggregation for further analysis
Taking ownership of data storage and management within AWS, including the separation of data into different buckets and zones
Working closely with IT administrators to ensure proper access control and data security
Implementing data tokenization mechanisms to secure sensitive information, ensuring compliance with data privacy regulations
Continuously monitoring the Data Lakes performance and identifying opportunities for improvement
Optimizing data processing pipelines and storage structures to enhance data retrieval speed and overall system efficiency
Working closely with the Product Group, who are the data owners, to understand their data needs, provide technical guidance, and ensure data quality and integrity throughout the data lifecycle
Collaborating with data scientists to ensure data availability meets business requirements
Providing expert technical support to the Data Lake team and data users, addressing issues related to data ingestion, processing, and access
Troubleshooting and resolving any data-related problems in a timely manner
Promoting best practices in data governance, ensuring adherence to data standards and documentation requirements
Maintaining comprehensive documentation of data processing workflows and data lineage for future reference
requirements-expected :
At least 5+ years of experience as a Data Engineer, with a focus on building and managing Data Lake infrastructures
Strong expertise in AWS services, including data storage and processing using S3, Databricks, and Spark
Hands-on experience with Databricks, Apache Spark, and Python for data processing and optimization
Proficiency in working with big data formats like Parquet, Iceberg, Delta Format
Solid understanding of data privacy regulations and best practices in handling PII data
Experience with data visualization and reporting tools like Qlik Sense is a plus
Strong problem-solving skills, attention to detail, and the ability to work in a collaborative team environment
Excellent communication skills, with the ability to convey technical concepts to both technical and non-technical stakeholders.
Experience with containerization technologies and Infrastructure as Code (IaC) using Terraform
Proficiency with orchestration tools like Airflow
Proficiency in English at the C1 level
offered :
Financial transparency: the salary is: 26 880 – 33 600 PLN (160 – 200 PLN/h) + VAT
Clear wage model. 7N Consultants know exactly what’s 7N fee for its agent role on the project
Professional development support. We subsidize trainings, technical certificates, conference participation, and foreign language classes. Additionally, 7N Consultants have opportunities to take part in internal training sessions and develop their interpersonal skills within the framework of the 7N Secret Code
Access to e-learning platforms: internal – 7N Academy with recordings of technical trainings (e.g. AWS, Azure, AI, DevOps, Data Science, etc.), inspiring webinars (e.g. healthy eating, personal finance, language learning, etc.), and external platforms: Coursea, Mentorix
Cloud Path - a development program that includes online trainings on AWS and Azure, ending in a certification exam. Each participant receives access to e-learning platforms: Microsoft ESI, Google Cloud Platform
An opportunity to join 7N Inspiration Team - a group of 7N Consultants who consistently and actively share their professional knowledge and expertise
Career Development Program – a mentoring program that includes exploring your individual predispositions and preferences for career paths; providing insight into the latest trends and valuable positions in the IT industry. Participants meet with Career Agents and/or 7N Mentors
Cooperation with real experts – the average professional experience of a 7N consultant is 10 years
Comprehensive and personal project support from one of a 7N Agent. Priority regarding project continuity and quality
Numerous (10-12 each year) high-end events – both online and offline
Healthcare, Benefit Multisport and life insurance subsidies regardless of cooperation model. Access to Mindgram, a psychological support and professional development platform
Professional and quality-oriented recruitment process led by the most experienced recruiters in the industry
benefits :
sharing the costs of sports activities
private medical care
sharing the costs of foreign language classes
sharing the costs of professional training & courses
