Senior Data Engineer

Knowledge & Experience Required

Essential Technical Skills

• Azure Databricks (Spark, Delta Lake, performance tuning).
• Python and PySpark for large-scale data processing.
• SQL (advanced querying, optimisation, and data modelling).
• Azure Data Factory (pipeline orchestration and integration).
• dbt (analytics engineering best practices).
• Azure DevOps (Git, CI/CD pipelines, release management).
• Lakehouse architecture (Databricks Unity Catalog, Delta Lake optimization techniques such as Z-ordering, liquid clustering)
• Data modelling (star schemas, data vault, or lakehouse-aligned approaches).
• Data quality, testing frameworks, and monitoring/observability.
• Strong problem-solving ability and a pragmatic, engineering-led mindset.
• Experience in Agile SW development environment
• Excellent communication skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
• Leadership and mentoring capability, with a focus on raising engineering standards and best practices.

Essential Experience

• Significant commercial experience (typically 5+ years) in data engineering roles, with demonstrable experience designing and operating production-grade data platforms.
• Strong hands-on experience with Azure Databricks, including cluster configuration, job orchestration, and performance optimisation.
• Proven experience building data pipelines with Databricks and Azure Data Factory; integrating with Azure-native services (e.g. Data Lake Storage Gen2, Azure Functions).
• Advanced experience with Python for data engineering, including PySpark for distributed data processing.
• Strong SQL expertise, with experience designing and optimising complex analytical queries and data models.
• Practical experience using dbt in a production environment, including model design, testing, documentation, and deployment.
• Experience implementing CI/CD pipelines using Azure DevOps or equivalent tooling.
• Data as a Product mindset

AI / ML / GenAI Enablement

• Enable ML/AI workloads on the Databricks data platform
• Support Databricks AI capabilities (e.g., Agent Bricks, Genie A/B etc)
• Collaborate with AI Product Team to deliver use cases
• Enable RAG pipelines / vector storage patterns to support AI products

Desired Experience

• Financial services industry or private market experience
• Development with coding agents (e.g., Anthropic Claude Code, OpenAI Codex etc)

#LI-SB1

This job description is not to be construed as an exhaustive statement of duties, responsibilities, or requirements. You may be required to perform other job-related duties as reasonably requested by your manager.

Pantheon is an Equal Opportunities employer, we are committed to building a diverse and inclusive workforce so if you're excited about this role but your past experience doesn't perfectly align we'd still encourage you to apply.