Data Engineer Ab initio

Job Title: - Data Engineer Ab initio - Ab Initio,
Location: Berkley Heights, NJDuration - Longterm Role Overview:

• We are seeking a highly skilled resource to design and implement high-performance, event-driven data
• pipelines, ensuring low-latency data processing and high availability system for the large credit card
• processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to
• build stateful services that ingest, filter, and transform data from sources like Kafka or message queues,
• pushing updates to dashboards or downstream databases in near-real-time.

Key Responsibilities:

• Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and
• aggregations.
• Implement event-driven data pipelines using Kafka, MQ, and file streams.
• Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-
• once" processing.
• Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency
• performance.
• Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues
• Develop ETL pipelines for batch and real-time data ingestion and transformation.
• Implement and ensure data validation, data security, integrity, and compliance across big data
• platforms.
• Monitor and troubleshoot performance issues in large-scale clusters.
• Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.
• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills & Qualifications:

• Deep understanding of credit card process system
• Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment),
• Conduct>It, and Continuous Flows.
• Understanding of Kafka, message queues, and real-time stateful services.
• Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata).
• Experience in Java , Scala, Python or Kafka is plus.
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of data security, governance, and compliance standards.
• Experience with cloud-based big data platforms
• Exposure to containerization (Docker, Kubernetes) for big data workloads.
• Knowledge of CI/CD pipelines for data engineering projects.

Behavioral Skills:

• Good Communication skills
• 5 days Work from Office at Berkley Heights, NJ
• Team Player
• Ability to work in a changing environment
• Strong problem solving and analytical skills
• Ability to work independently or within a team
• Manage day-to-day challenges and communicate developmental risks with the technical team