Senior Platform Engineer

Job Description

Databricks and Azure - Platform Engineering and Developer Enablement

As part of our client's engineering organization, we are looking for a Senior Platform Engineer with deep expertise in Databricks platform operations, automation and cloud engineering. This role enables and scales enterprise-grade data platforms through automation, governance, infrastructure and developer experience improvements.

This is not a traditional Data Engineer position. The successful candidate must understand Databricks data engineering concepts, pipelines, notebooks and platform architecture, but their primary responsibility is to build and maintain the platform capabilities, automation frameworks and engineering tooling that support multiple product teams operating at scale.

The role is distinct from the pod-embedded DevOps Engineer. The Platform Engineer builds the paved road that pods consume: workspace automation, catalogue governance and self-service capability. The DevOps Engineer owns a single pod's pipelines, environments and runtime on top of that road. Candidates should be clear which of the two they are applying for.

Key Responsibilities:
- Design, develop and maintain platform services and automation capabilities supporting Databricks environments.
- Build and enhance platform tooling that simplifies the deployment, configuration, governance and operation of Databricks workspaces.
- Develop reusable engineering capabilities that accelerate onboarding of new teams and products onto the client's data platform.
- Partner with architects, data engineers and product teams to define and implement platform standards and best practices.
- Automate provisioning, configuration management, environment setup, monitoring, security and operational workflows.
- Support CI/CD processes for platform and infrastructure components.
- Troubleshoot and resolve complex issues across Databricks, Azure, GitHub and platform integrations.
- Review and help debug notebooks, jobs, workflows and pipeline implementations when platform expertise is required.
- Contribute to platform architecture decisions and long-term engineering strategy.
- Improve platform reliability, scalability, security, observability and operational excellence.
- Participate in design reviews, technical governance and engineering standards development.

About You:
The ideal candidate is a senior platform engineer who understands how Databricks works under the hood and is comfortable debugging platform issues, automating operations and improving the engineering experience for product teams. They are equally comfortable discussing cloud architecture, CI/CD pipelines, infrastructure automation and software development. They are not primarily a notebook developer or pipeline builder. Instead, they focus on creating the platform foundations that enable others to build and operate data solutions efficiently, securely and at scale.

Requirements

• Strong hands-on experience with Databricks in enterprise environments.
• Deep understanding of workspaces, clusters, jobs, Unity Catalog, security and permissions, workspace configuration, environment management, monitoring and operations.
• Ability to troubleshoot and support data pipelines and platform services.
• Understanding of data engineering workflows without being limited to pipeline development.
• Strong experience designing and operating solutions on Microsoft Azure.
• Knowledge of Azure networking, identity and access management, storage services, monitoring and observability, infrastructure automation, security and governance controls.
• Experience working within large-scale enterprise cloud environments.
• Strong development experience in at least one modern programming language such as Python, Java, C# or TypeScript.
• Ability to design and build maintainable platform services and automation frameworks.
• Experience with API integrations and platform extensions.
• Strong CI/CD background and experience creating and maintaining automated deployment pipelines.
• Infrastructure-as-code experience using Terraform or similar technologies.
• Experience with Git-based development workflows.
• Strong GitHub experience including Actions, repository management, branching strategies, pull request workflows and automation.
• Experience working in Git-based development models across multiple teams.
• Experience operating large-scale enterprise data platforms.
• Understanding of platform governance, security, scalability, performance and operational processes.
• Experience supporting multiple teams and products within a shared platform ecosystem.
• Practical experience directing AI coding agents such as Factory.ai or GitHub Copilot to produce production code, and reviewing that output critically rather than accepting it.
• Ability to write precise, testable specifications that an agent can build from without coming back for clarification.
• Sound judgment about where agent autonomy is appropriate and where a human decision is mandatory.
• Comfort being measured on agent autonomy coverage and rework rate alongside conventional delivery measures.
• Preferred: Experience with enterprise data platforms supporting multiple product teams and a broad user base.
• Preferred: Experience integrating cloud-native services with Databricks.
• Preferred: Experience building developer self-service capabilities and platform engineering solutions.
• Preferred: Exposure to AI and ML platform operations and governance.
• Preferred: Knowledge of observability, monitoring and platform reliability practices.
• Preferred: Experience working in global, distributed engineering organizations.

Benefits

• Pods of five with end-to-end ownership. Your pod builds, tests, releases and runs its own slice of the platform. Nothing is handed to a separate QA or operations function.
• Kanban and flow rather than sprints. Work is pulled, work in progress is limited, and throughput, lead time and cycle time are measured. There is no sprint commitment ceremony and no PI planning.
• A six-phase product lifecycle: Plan, Define, Build, Review, Release and Support.
• Tooling: GitHub for Projects, Issues, Actions and GitOps; Factory.ai and GitHub Copilot for agentic engineering; Nullplatform for environments; Aha for portfolio intake.
• Agent-first delivery. We target 60 to 70 percent agent autonomy on eligible work and rework below 15 percent, which shifts engineering effort from typing code to specifying and reviewing it.