About the Role
We are looking for a detail-oriented and analytical AI QA Tester to bridge the gap between traditional software quality assurance and probabilistic machine learning systems. In this role, you will test LLMs, AI agents, RAG (Retrieval-Augmented Generation) pipelines, and intelligent features to ensure they meet high standards for factual accuracy, safety, performance, and user trust.
Key Responsibilities
- AI & LLM Evaluation: Design and execute test cases to evaluate model responses for hallucinations, factual correctness, relevance, and completeness.
- Safety & Guardrail Testing: Conduct security and adversarial testing—including prompt injection, jailbreaking, and bias/fairness checks—to verify system resilience against malicious or skewed inputs.
- Regression & Golden Datasets: Build, maintain, and run "golden datasets" to benchmark new model versions, prompts, or pipeline updates against expected behavior.
- Traditional QA Integration: Combine AI-specific evaluations with core QA methodologies, including functional testing, API validation (Postman/Swagger), regression suites, and bug tracking (Jira).
- Performance & Scalability: Simulate high-concurrency user loads to measure AI response latency, token usage, timeouts, and system stability.
- Defect Reporting & Collaboration: Work closely with ML engineers and product managers to document edge cases, track failures, and define clear acceptance criteria for AI features.
Qualifications & Requirements
- Experience: [2+] years of experience in software quality assurance, test automation, or technical product evaluation.
- AI/ML Familiarity: Solid understanding of how Large Language Models (LLMs), RAG workflows, and AI agents operate.
- Technical Skills:
- Proficiency in Python or SQL for data analysis and test scripting.
- Experience with testing frameworks or evaluation tools (e.g., PyTest, DeepEval, RAGAS, or similar).
- Familiarity with API testing and CI/CD pipelines (GitHub Actions, Azure DevOps).
- Mindset: Strong critical thinking and comfort dealing with probabilistic outputs rather than strict binary pass/fail logic.
Nice-to-Have
- Experience building or evaluating RAG/agentic applications from scratch.
- Knowledge of data drift detection, compliance logging, and AI governance frameworks.
Analytical and detail-oriented AI Test Engineer with 4+ years of experience designing validation pipelines and quality frameworks for machine learning models and RAG (Retrieval-Augmented Generation) systems. Skilled in Python, PyTest, RAGAS, and LLM evaluation metrics (hallucination rate, factual accuracy, and data drift detection). Proven track record of reducing model hallucinations by 35% and building automated regression suites for AI-driven enterprise applications.
Core Responsibilities List
- Test Strategy & Evaluation Design:
- Plan non-deterministic test approaches where outputs vary across runs rather than matching a single fixed string.
- Design evaluation rubrics, golden datasets, and automated LLM-as-a-judge frameworks to measure output quality.
- Model & Pipeline Validation:
- Validate training and inference pipelines, data ingestion, preprocessing, and vector retrieval (RAG) workflows.
- Test APIs serving models for latency, scalability, and failure recovery.
- Safety, Red-Teaming & Guardrails:
- Conduct red-teaming to test for prompt injections, jailbreaks, data leakage, and toxic or biased outputs.
- Evaluate edge cases, hallucinations, and confidence thresholds.
- AI Agent & Tool Testing:
- Test multi-step agent loops, decision-making, tool/API selection, and handling of infinite loops or unauthorized actions.
- Regression & Post-Deployment Monitoring:
- Build regression test suites for new model versions, prompt changes, or updated retrieval strategies.
- Monitor data drift, concept drift, and performance degradation in production.
- Cross-Functional Collaboration:
- Work with data scientists, ML engineers, and product owners to translate business logic into measurable AI quality metrics.
#TOP_POSITION
