93 902
15 856
Technologies-expected : AWS S3 Databricks Spark Apache Spark Python technologies-optional : Qlik Sense Terraform Airflow about-project : We are seeking two highly skilled and experienced Data Engineers who will join a team working in a global container liner shipping company. Team is specialising in building Data Lake and leveraging the Databricks platform. Possibility of 100% remote work. responsibilities : Designing and implementing robust data processing pipelines to efficiently ingest, integrate, and transform raw data from diverse sources such as internal databases, Io T data, fleet data, and third-party sources Ensuring seamless integration of data into the Data Lake, adhering to data quality and consistency standards Utilizing your strong expertise in Databricks and Apache Spark to develop and optimize data processing workflows within the Data Lake Leveraging Sparks capabilities to process data at scale, perform complex transformations, and facilitate data aggregation for further analysis Taking ownership of data storage and management within AWS, including the separation of data into different buckets and zones Working closely with IT administrators to ensure proper access control and data security Implementing data tokenization mechanisms to secure sensitive information, ensuring compliance with data privacy regulations Continuously monitoring the Data Lakes performance and identifying opportunities for improvement Optimizing data processing pipelines and storage structures to enhance data retrieval speed and overall system efficiency Working closely with the Product Group, who are the data owners, to understand their data needs, provide technical guidance, and ensure data quality and integrity throughout the data lifecycle Collaborating with data scientists to ensure data availability meets business requirements Providing expert technical support to the Data Lake team and data users, addressing issues related to data ingestion, processing, and access Troubleshooting and resolving any data-related problems in a timely manner Promoting best practices in data governance, ensuring adherence to data standards and documentation requirements Maintaining comprehensive documentation of data processing workflows and data lineage for future reference requirements-expected : At least 5+ years of experience as a Data Engineer, with a focus on building and managing Data Lake infrastructures Strong expertise in AWS services, including data storage and processing using S3, Databricks, and Spark Hands-on experience with Databricks, Apache Spark, and Python for data processing and optimization Proficiency in working with big data formats like Parquet, Iceberg, Delta Format Solid understanding of data privacy regulations and best practices in handling PII data Experience with data visualization and reporting tools like Qlik Sense is a plus Strong problem-solving skills, attention to detail, and the ability to work in a collaborative team environment Excellent communication skills, with the ability to convey technical concepts to both technical and non-technical stakeholders. Experience with containerization technologies and Infrastructure as Code (Ia C) using Terraform Proficiency with orchestration tools like Airflow Proficiency in English at the C1 level offered : Financial transparency: the salary is: 26 880 – 33 600 PLN (160 – 200 PLN/h) + VAT Clear wage model. 7 N Consultants know exactly what’s 7 N fee for its agent role on the project Professional development support. We subsidize trainings, technical certificates, conference participation, and foreign language classes. Additionally, 7 N Consultants have opportunities to take part in internal training sessions and develop their interpersonal skills within the framework of the 7 N Secret Code Access to e-learning platforms: internal – 7 N Academy with recordings of technical trainings (e.g. AWS, Azure, AI, Dev Ops, Data Science, etc.), inspiring webinars (e.g. healthy eating, personal finance, language learning, etc.), and external platforms: Coursea, Mentorix Cloud Path - a development program that includes online trainings on AWS and Azure, ending in a certification exam. Each participant receives access to e-learning platforms: Microsoft ESI, Google Cloud Platform An opportunity to join 7 N Inspiration Team - a group of 7 N Consultants who consistently and actively share their professional knowledge and expertise Career Development Program – a mentoring program that includes exploring your individual predispositions and preferences for career paths; providing insight into the latest trends and valuable positions in the IT industry. Participants meet with Career Agents and/or 7 N Mentors Cooperation with real experts – the average professional experience of a 7 N consultant is 10 years Comprehensive and personal project support from one of a 7 N Agent. Priority regarding project continuity and quality Numerous (10-12 each year) high-end events – both online and offline Healthcare, Benefit Multisport and life insurance subsidies regardless of cooperation model. Access to Mindgram, a psychological support and professional development platform Professional and quality-oriented recruitment process led by the most experienced recruiters in the industry benefits : sharing the costs of sports activities private medical care sharing the costs of foreign language classes sharing the costs of professional training & courses life insurance integration events employee referral program access to internal trainings – 7 N Academy Cloud Path – cloud development program access to external e-learning platforms Microsoft ESI Google Cloud Platform Coursea Mentorix career development program technical training inspirational training numerous high-end events access to the Mindgram platform 7 N Inspiration Team