Senior Databricks Data Engineer

Mandatory Skills: Databricks, Data Engineering, Lakeflow, Unity Catalogue, Python, PySpark, ETL.

About the Role:

We are looking for an experienced Senior Databricks Data Engineer with strong expertise in Databricks, SQL, Lakeflow Jobs, PySpark notebooks, Python, and Unity Catalogue. This role requires hands-on experience in designing scalable data pipelines and delivering reliable data solutions on the Databricks Lakehouse platform.

Key Responsibilities:

• Design, develop, and optimize data pipelines using Databricks, Python, PySpark, and Spark SQL.
• Orchestrate and schedule workflow jobs that invoke notebooks, Python wheel packages, and Python notebooks.
• Work with Unity Catalog, manage tables, and streaming jobs.
• Implement and manage data ingestion, transformation, and loading processes from various sources into the Databricks Lakehouse.
• Build and maintain efficient and reliable ETL/ELT processes to ensure data quality and availability.
• Optimise Databricks workloads for performance, cost efficiency, and scalability.
• Troubleshoot and resolve data-related issues while ensuring data integrity and accuracy.
• Stay current with the latest Databricks features, best practices, and industry trends.

Required skills & Qualifications:

• At least 4 years of proven experience as a Data Engineer with a strong focus on Databricks (mandatory).
• Strong knowledge of Databricks architecture, Lakeflow Jobs, and data modelling.
• At least 10 years of experience in the data domain, particularly in data warehousing and ETL (mandatory).
• Expertise in PySpark, Spark SQL, and Python programming (mandatory).
• Hands-on experience with Git, including code check-in and check-out processes (mandatory).
• Strong understanding of data warehousing, data lakes, and Lakehouse architecture.
• Databricks Certified Data Engineer Associate or Professional certification is an advantage.

Soft Skills:

• Strong communication and stakeholder collaboration skills.
• Proactive, self-driven, and accountable mindset.
• Excellent analytical and problem-solving abilities.
• Ability to work effectively in Agile and cross-functional teams.
• Customer-focused with a commitment to quality delivery.
• Ability to converse seamlessly in English and one of either Dutch or French or both