Lead Data Engineer

Job Summary (Lead Data Engineer Data Transformation & Consumption)

- Lead and drive data modernization, transformation, governance, and analytics consumption initiatives.

- Build and manage scalable ETL/ELT pipelines for diverse data types (structured, semi-structured, unstructured).

- Develop batch and real-time data processing solutions using modern cloud and big data technologies.

- Design and implement data models, data marts, and analytics-ready datasets for BI, AI, and ML.

- Lead cloud data engineering projects across AWS, Azure, or GCP.

- Ensure data quality, governance, lineage, security, and compliance throughout the data ecosystem.

- Collaborate with business stakeholders, data architects, data scientists, and BI teams to deliver data-driven business value.

- Modernize legacy data warehouses and support data lakehouse architectures and platform transformations.

- Establish and enforce engineering best practices, CI/CD pipelines, Infrastructure as Code (IaC), and automation.

- Mentor and lead data engineering teams; conduct architecture reviews and provide technical governance.

- Optimize platform performance, scalability, reliability, and manage cloud costs.

Core Technical Skills

- Proficient in: SQL, Python, PySpark, Spark

- Data modeling (Star Schema, Snowflake, Data Vault)

- ETL/ELT tools: Airflow, DataStage, Informatica, Talend, DBT

- Cloud data warehouses: BigQuery, Snowflake, Synapse, Redshift

- Event streaming: Kafka, Pub/Sub, Event Hub

- DevOps: Git, CI/CD, Terraform, Docker, Kubernetes

- Cloud platforms: AWS, Azure, GCP

Preferred Skills

- Experience with Data Lakehouse (Databricks, Delta Lake)

- Master Data Management (MDM)

- Data Mesh/Data Product Architecture

- AI/ML & Generative AI data platforms

- Domain experience in BFSI, Insurance, Retail, or Healthcare