Informacje o stanowisku

Mindverse Consulting Services | Full time

Data Engineer with extensive Scala experience | 2024TA1201x110402

• Duration 02.01.2025 – 30.06.2025 | 6-12 Months
• Date Opened 21/02/2025
• Last Date of Application ASAP
• Job Type Full time
• Experience Band 7 Years
• Positions 5
• Location Remote | Anywhere in Poland only
• Salary Quote your expected Hourly Rate in EUR
• Job Opening Status In-progress
• City Warsaw
• Country Poland
• Postal Code 00-002

About Us

We are an SME IT Consulting Company focusing on Cloud Computing & Data Science Consulting across Europe and India .

We support our customers by providing technical consultants on contract as per their project requirements.

We refer to such services as SKILL-PARTNERSHIP and are operational in 40+ countries globally .

Job Description

Our Customer is looking for Data Engineers who have extensive experience with Scala.

Main Responsibilities:

• Data Pipeline Maintenance: Continuously monitor and maintain data pipelines for ingesting and transforming data using Scala and SQL on Spark. Diagnose and resolve errors and performance bottlenecks, addressing data discrepancies, ambiguities, and inconsistencies as needed.
• Technical Support and Version Control: Provide technical support for data analysis while managing source code and configuration artifacts via GitHub. Deploy code artifacts through GitHub Workflows/Actions.
• Technical Leadership: Offer hands-on technical guidance and leadership in developing Spark-based data processing applications using Scala, focusing on Microsoft Azure Synapse Spark Runtime.
• Pipeline Optimization: Design and enhance data pipelines to streamline processing across various stages of the Medallion architecture using Azure Synapse Pipelines.
• Data Management: Oversee data ingestion processes, enforce data quality checks using tools like DQ, and manage validation and error-handling workflows.
• Configuration Management: Develop and manage configuration settings using JSON-based configurations (e.g., ApplicationConfig, TableConfig) for multiple data zones.
• Collaboration: Work closely with data scientists, analysts, and cross-functional teams to ensure smooth integration of data engineering efforts with marketing and business strategies.
• Logging and Auditing: Implement and manage logging, auditing, and error-handling practices to maintain data processing integrity, leveraging tools like Azure Log Analytics and KQL queries where applicable.
• Testing and Quality Assurance: Conduct unit testing with tools like ScalaTest and maintain rigorous data quality checks to ensure dependable processing outcomes.

Technical Requirements:

• 7+ years of professional experience in Data Engineering.
• Minimum 6-7 years of experience in the Data Engineering space on Scala & Spark + Azure.
• Strong proficiency in programming languages such as Python and Scala.
• Hands-on experience with data pipeline tools.
• Expertise in relational and non-relational databases, including SQL.
• Solid knowledge of cloud platforms and their data services.
• Familiarity with containerization and orchestration (e.g., Docker, Kubernetes) is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced environment.
• Excellent communication and collaboration abilities.

Warszawa - Oferty pracy w okolicznych lokalizacjach