AI Quality Engineer (GenAI / RAG / Agentic AI) - Pheonix, AZ - Contract
Is this the right opportunity for you?
Explore more jobs, compare advertised salaries and see which skills employers want.
Explore this careerSearch for a different role
More Quality Inspector jobs →Skills
Description
We at NJ TECH INC are focused on hiring highly skilled professionals who are excited by the opportunity to make a true impact on their careers as well as on our clients' business. We power our clients’ success—and drive our consultants’ career growth.
We are seeking an experienced and outstanding AI Quality Engineer (GenAI / RAG / Agentic AI) for one of our esteemed clients.
Role: AI Quality Engineer (GenAI / RAG / Agentic AI)
Work Location: Pheonix, AZ
Work Mode: Onsite
Employment Type: Long-Term Contract
Job Summary
We are seeking a hands-on AI Quality Engineer to design and implement quality-engineering strategies for Generative AI, Retrieval-Augmented Generation (RAG), and Agentic AI applications.
The ideal candidate will have strong Python automation skills and hands-on experience evaluating AI applications to ensure accuracy, reliability, safety, performance, and production readiness. The role involves building automated evaluation frameworks, establishing quality benchmarks, and integrating AI testing into CI/CD pipelines.
Key Responsibilities
Design and implement end-to-end testing strategies for LLM, RAG, and Agentic AI applications.
Build automated evaluation frameworks to measure answer correctness, relevance, groundedness, faithfulness, hallucination, retrieval quality, tool selection, and agent task completion.
Develop and maintain golden datasets, benchmark suites, and prompt/model regression test packs.
Test RAG pipelines covering document ingestion, chunking, embeddings, vector retrieval, reranking, context assembly, and response generation.
Validate multi-step agent workflows, including planning, reasoning, tool execution, memory, exception handling, guardrails, and human-in-the-loop controls.
Perform negative, boundary, adversarial, prompt-injection, jailbreak, and sensitive-data leakage testing.
Validate APIs, microservices, enterprise integrations, and downstream systems accessed by AI agents.
Automate functional, API, integration, and end-to-end testing using Python, PyTest, Playwright, Selenium, and REST API frameworks.
Integrate AI evaluations and automated tests into CI/CD pipelines with actionable reporting and quality gates.
Monitor production quality across accuracy, latency, reliability, token consumption, and cost.
Identify and troubleshoot failures across prompts, retrieval, data, models, orchestration, tools, and application code.
Required Skills
AI / GenAI Quality Engineering
Strong understanding of LLMs, Generative AI, RAG, embeddings, vector search, and Agentic AI.
Experience evaluating non-deterministic AI outputs and defining acceptance thresholds.
Knowledge of RAG evaluation, prompt regression, hallucination detection, grounding, and agent/tool-call validation.
Ability to create representative golden datasets and benchmark suites.
Automation Engineering
Strong hands-on experience with Python and PyTest.
REST API testing and automation using Playwright or Selenium.
Experience building reusable test automation frameworks.
Experience integrating automated tests with Git-based CI/CD pipelines.
Working knowledge of SQL, microservices, Docker, and Kubernetes.
AI Evaluation & Observability
Hands-on exposure to DeepEval, RAGAS, Giskard, LangSmith, Langfuse, or custom Python evaluation frameworks.
Understanding of LLM-as-a-Judge, deterministic checks, human evaluation, tracing, and production observability.
Cloud & Platform
Experience with Azure OpenAI / Azure AI Foundry, Amazon Bedrock, or Google Vertex AI.
Working knowledge of vector databases, model endpoints, identity and access controls, secrets management, and cloud monitoring.
Quality Metrics
Response Quality: Answer relevance, correctness, faithfulness, groundedness, and hallucination rate.
RAG Quality: Context precision/recall, Retrieval Precision@K and Recall@K, and citation accuracy.
Agent Quality: Task-success rate, tool-call accuracy, workflow completion, and recovery rate.
Non-functional: P95 latency, availability, throughput, token usage, and cost per transaction.
Safety: Prompt-injection resistance, jailbreak success rate, toxicity, bias, and PII leakage.
Regression: Automated pass rate, model/prompt variance, and escaped defect rate.
Preferred Qualifications
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science, or a related discipline.
Experience testing production GenAI or RAG applications in regulated enterprise environments such as banking, insurance, or healthcare.
Working knowledge of LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, or comparable orchestration frameworks.
Exposure to Responsible AI, model risk management, security testing, and AI governance.
Relevant cloud, testing, security, or AI certifications.
Success Measures
Reduced AI defects escaping into production.
Measurable improvement in grounding, retrieval, and answer quality.
High automated regression coverage across prompts, models, RAG pipelines, and agent workflows.
Reliable AI quality gates embedded in delivery pipelines.
Reduced manual validation effort and faster release cycles without compromising safety or reliability.
Ideal Candidate Profile
A strong quality-engineering and automation professional with hands-on GenAI/RAG knowledge, advanced Python skills, and demonstrated ability to measure AI quality programmatically and diagnose failures across the complete AI application stack.
NJ TECH INC is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, We cannot sponsor any visa at this time
NJ TECH INC, is a globally managed IT service, consulting and Business solutions partner. Our "High-Performance Business" strategy builds up our expertise in technology and consulting. Our consulting plays a major role in helping clients to achieve their objectives in the highest level; ultimately creating sustainable value for customers. Come, transform your career with us and be a part of our high-performing team
Get similar jobs in United States by email
We'll email you when new jobs similar to this one appear.
Similar jobs
Explore more Quality Inspector jobs in United States.
Finding similar jobs…