Data Engineer- Databricks/ SQL/ Python/ PySpark

Data Engineer- Databricks: SQL/ Python/ PySpark

Location: London (Hybrid: 3 days travel to the office in Central London per week + 2 days’ work from Home)

Salary- up to £56K base + pension + healthcare.

Data Engineer required for one of our global clients in the UK for their BI & Analytics Team. You will be working with the leading tools and platforms such as Power Platform, Databricks, ADF.

Duties & Responsibilities

• Data Pipeline development/orchestration/maintenance
• Creating robust, fit-for-purpose data modelling solutions to address multiple business cases
• Root cause analysis and pro-active investigation to identify potential issues and optimize ETL pipelines, notify end-users and propose adequate solutions
• Prepare documentation for further reference

Software Knowledge

• Demonstratable delivery experience using Azure Databricks
• Solid delivery experience using ADF, ADLS Gen2, and the Azure Cloud Ecosystem
• Solid SQL skills – Querying syntax, DDL and DML statements
• Solid Python skills – able to use the Python programming concepts in Databricks Notebooks for data loading, transformation and exploration including familiarity with Pandas
• Strong understanding of core Databricks and Spark concepts such as Delta Live Tables, Delta Sharing, Medallion Architecture, Dataframes, Workflows, Unity Catalogue, Delta Live Tables, and UDF’s
• Understands the key principles of Data Warehouse design including Star Schema, F&D design, Dimension and Fact loading patterns, SCD and CDC concepts
• Experience with use of MS DevOps for Scrum management and with Repos and use of CI / CD Pipelines between Azure and DevOps
• Proven track record in a data engineering role where the focus has been on developing data pipelines, data warehouses, data lakehouses, and similar data repositories for BI, Reporting and Analytics use

Nice to have

• Any relevant Azure certifications / exams such as: AZ-900, DP-900, DP-203, or DP-500
• Any relevant Databricks certifications: Lakehouse Data Engineer Associate, : Lakehouse Data Engineer Professional