Senior Data Engineer

We are looking for an experienced Senior Data Engineer to design, develop, and maintain scalable enterprise data platforms and data products. The ideal candidate should have strong expertise in Python, SQL, Apache Spark, Azure Databricks, and cloud technologies across Azure and GCP.

Key Responsibilities:

• Design and develop scalable data ingestion and transformation pipelines.
• Build and optimize data solutions using PySpark, Python, SQL, and Databricks.
• Implement data governance, data quality, security, and lineage frameworks.
• Drive data platform architecture, performance tuning, and cost optimization.
• Develop CI/CD pipelines and Infrastructure as Code using Terraform/ARM.
• Mentor junior engineers and establish coding standards and best practices.
• Explore emerging technologies and leverage AI-assisted coding tools to accelerate development.

Required Skills:

• Strong experience in Python (PySpark, Pandas) and advanced SQL.
• Hands-on experience with Apache Spark 3.3+ and Azure Databricks.
• Experience with ADLS Gen2, Azure Data Factory, BigQuery, GCS/BigLake, and Dataproc.
• Good knowledge of Azure and GCP cloud services, IAM, security, and monitoring.
• Experience with Terraform, CI/CD, DevOps, and data quality frameworks.
• Hands-on experience with AI tools and AI-assisted development.