We are seeking an advanced ML Ops Engineer to design and implement the infrastructure required to host, orchestrate, and manage up to 1,500 ML scoring processes within a new Databricks environment. The focus of the role is on operationalizing the ML scoring pipelines by setting up a scalable, secure, and well-monitored platform for data science teams to deploy their models.

The successful candidate will create the ML operational backbone that allows data scientists to run, monitor, and manage high-volume ML scoring efficiently.

Its hybrid role which requires 3x/week work from office.

AI & MLOps Engineer

Your responsibilities

• Set up Databricks clusters, jobs, and workflows for large-scale ML scoring use cases.
• Infrastructure as Code is used for reproducibility and governance (e.g., Terraform).
• Implement scalable infrastructure capable of running thousands of ML scoring tasks.
• Configure job scheduling, parallel execution strategies, and resource optimization.
• Monitoring and alerting are integrated into the platform using cloud-native tools.
• Security, compliance, and cost-efficiency are key pillars of the operational setup.
• Develop deployment processes for ML models using Databricks MLflow or equivalent.
• Implement version control and tracking for models, scoring code, and configuration files.
• Integrate logging, alerting, and dashboards to monitor scoring throughput, latency, and failures.
• Establish model performance monitoring hooks for post-scoring analytics.
• Work alongside Dev Ops Engineers to ensure common infrastructure and processes (e.g., shared storage, Delta Lake tables) serve both ML and BI use cases.
• Automate provisioning of resources and deployments from CI/CD pipelines
• Utilize Infrastructure as Code (IaC) where feasible for reproducibility

Our requirements

• Proven experience with ML Ops in production ML environments.
• Strong hands-on knowledge of Databricks (MLflow, Jobs, Workflows, Delta Lake).
• Experience with large-scale batch job orchestration and distributed computing.
• Familiarity with Python for workflow scripting and pipeline integration.
• Experience in CI/CD pipelines for ML model deployment (Azure DevOps, GitHub Actions, or similar).
• Proficiency with monitoring tools, logging frameworks (e.g., Datadog, Prometheus, Grafana, or built-in cloud monitoring).
• Understanding of Infrastructure as Code and cloud environment automation.
• Knowledge of model lifecycle management, versioning, and reproducibility.
• English Level: B2
• Experience executing high-volume ML scoring in Databricks.
• Familiarity with industry best practices for ML operationalisation in regulated environments.
• Knowledge of job queueing systems and parallel execution patterns for ML workloads.
• Exposure to Azure Databricks and the Azure ecosystem.
• Experience with performance tuning for large concurrent workloads.
• Awareness of cost optimisation for always-on ML scoring infrastructure.