This role involves designing and developing Databricks applications in pySpark. Our team is rewriting an existing SQL on-prem data warehouse into a Data Lakehouse using Databricks.
responsibilities :
Migration from on-prem SQL Data Warehouse to Databricks.
Developing pySpark applications and Spark jobs.
Maintaining Databricks workspaces, clusters, and jobs.
Integrating Databricks applications with various technologies.
Keep Databricks environment healthy.
requirements-expected :
Must be a Subject Matter Expert in Spark.
Proficiency with Big Data processing technologies (Hadoop, Spark, Databricks).
Experience in building data pipelines and analysis tools using Python, pySpark, Scala.
Create Scala/Spark jobs for data transformation and aggregation.
Produce unit tests for Spark transformations and helper methods.
Design data processing pipelines.
Good to have experience with Hadoop / Databricks.
Passionate about learning new technologies.
Ability to learn new concepts and software quickly.
Analytical approach to problem-solving; ability to use technology to solve business problems.
Familiarity with database-centric applications.
Ability to communicate effectively in both a technical and non-technical manner.
Ability to work in a fast-paced environment.
Experience working in an agile environment using SCRUM methodology.
Communicate and interact with all levels of personnel within the organization, including senior management, and other departments.
Results oriented, team player with strong attention to detail.
Bachelor’s degree or relevant work experience in Computer Science, Mathematics, Data Engineering or related technical discipline.
benefits :
sharing the costs of sports activities
private medical care
sharing the costs of professional training & courses
life insurance
remote work opportunities
flexible working time
fruits
corporate products and services at discounted prices
