<p> </p> <p><strong>Job Title:</strong> Senior AI QA Automation Engineer</p> <p><strong>Key Skills:</strong> Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring</p> <p><strong>Experience:</strong> 5+ YOE.</p> <p><strong>Location:</strong> Costa Rica, Peru, Colombia, Bolivia.</p> <p><strong>Mode:</strong> Remote</p> <p>We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.</p> <p><strong>Key Responsibilities</strong></p> <ul> <li> <p>Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.</p> </li> <li> <p>Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.</p> </li> <li> <p>Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.</p> </li> <li> <p>Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.</p> </li> </ul> <p><strong>Required Skills & Qualifications</strong></p> <ul> <li> <p>5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.</p> </li> <li> <p>Strong Python skills with experience building reusable automated testing frameworks.</p> </li> <li> <p>Hands-on experience with Playwright and/or Selenium.</p> </li> <li> <p>Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.</p> </li> <li> <p>Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.</p> </li> <li> <p>Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.</p> </li> <li> <p>Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.</p> </li> <li> <p>Experience implementing CI/CD quality gates and production monitoring.</p> </li> <li> <p>Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.</p> </li> <li> <p>Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.</p> </li> </ul> <p><strong>Preferred Skills:</strong></p> <ul> <li> <p>Experience testing Generative AI, RAG solutions, copilots, or AI agents.</p> </li> <li> <p>Experience with LLM evaluation frameworks and advanced prompt testing.</p> </li> <li> <p>Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.</p> </li> <li> <p>Experience with adversarial testing and Responsible AI testing.</p> </li> <li> <p>Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.</p> </li> </ul> <p><strong>Posted On:</strong> 12-10-2026</p> <p>At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.</p>