BI Development Team is in the process of moving into Azure Cloud. One of the main areas is to administrate Spark Clusters (PaaS). Join us and be part of the Global BI Admin Team.
responsibilities :
Setup Synapse Linked Services and MS Synapse workspaces including Spark Clusters using Terraform
Spark version upgrades
Using Kafka, Oracle, etc.
Resource Management, Memory usage, optimizing CPU and resource usage for Spark Application
Launching and running executors on worker nodes
Monitor executor performance and scalability
Detecting handle worker node failures
Reallocating resources and restarting spark jobs
Monitoring the Spark cluster health, resource utilization, and job execution metrics
Monitoring by using Grafana and Prometheus
Performance tuning by fine-tuning Spark parameters
Working with Developers and helping to for improvement of code
IT Team Management
requirements-expected :
Min. 4 years of experience with Apache Spark: Advanced knowledge of Spark architecture, including components (RDDs, DataFrames, Datasets)
Very good knowledge of Spark SQL
Proficiency in setting up, configuring, and maintaining Spark clusters
Understanding of YARN
Ability to optimize Spark jobs, memory usage, and parallelism
Familiarity with monitoring tools (preferred: Grafana and Prometheus)
Hadoop: HDFS, Hive, and other related technologies
Proficiency in Python, pyspark and spark SQL
Knowledge of securing Spark clusters and data.
Ability to troubleshoot
Experience in managing an IT team
Good communication with developers, data engineers, and stakeholders.
Experience in cross-functional teams
offered :
Long-term cooperation based on our values: education, partnership and responsibility,
Work on projects dedicated to the worlds largest brands,
Possibility to exchange experiences in an international environment and develop projects implemented in Poland or other countries: Slovakia, Czech Republic, Bulgaria,
Working in a modern environment with innovative technologies,
Flexible working hours,
Private medical care,
Multisport or My Cafeteria card,
Life insurance,
NATEK loyalty Club,
Opportunity to join a socially responsible community and participate in company initiatives,
For contractors - entitlement to 23 additional days of service provision per year,
An additional paid day off for volunteering for salaried employees.
