<h2><strong>About the Role</strong></h2> <p>The<strong> Senior AI Evals Test Engineer</strong> is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the <strong>Google Agent Development Kit (ADK) or similar sort of agentic frameworks</strong>. This role bridges AI development and reliable deployment — ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus of the role is on Responsible AI and Safety Evals, including <strong>adversarial testing, red-teaming, bias detection, and toxicity measurement</strong>. The engineer will <strong>design scalable evaluation pipelines</strong>, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is <strong>70% Automation and 30% Manual Testing</strong>, and sits within the <strong>Applied AI: Test Engineering team</strong> as a key technical authority.</p> <h2><strong>Key Responsibilities</strong></h2> <ul> <li><strong>Design and build agent evaluation frameworks </strong>— create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate</li> <li><strong>Build synthetic testing environments and simulation strategies</strong> to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)</li> <li><strong>Build golden datasets and test harnesses</strong> — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)</li> <li><strong>Establish eval-as-a-service</strong> — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom benchmarks</li> <li><strong>Automate regression and drift detection</strong> — implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment</li> <li><strong>Define eval taxonomies and rubrics</strong> — establish structured scoring criteria for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)</li> <li><strong>Instrument production observability </strong>— build real-time monitoring for agent quality in production: success rates, user corrections, fallback triggers, escalation patterns</li> <li><strong>Benchmark and select models</strong> — run structured model comparisons (cost, quality, latency trade-offs) to inform model selection decisions across agent use cases</li> <li><strong>Analyze and interpret evaluation results</strong>, providing clear, actionable quality and safety reports to stakeholders and development teams, with specific focus on risk metrics, trend analysis, and mitigation recommendations</li> <li><strong>Define comprehensive end-to-end QA strategies </strong>— functional, integration, regression, and user acceptance testing (UAT) — tailored for conversational and goal-oriented AI agents with tool-use, memory, and multi-turn reasoning capabilities</li> <li><strong>Develop and maintain detailed test artifacts</strong> (test plans, test cases, test scenarios, automated test suites) covering agent features, tool invocation, memory persistence, and reasoning accuracy across evolving model versions</li> <li><strong>Identify, document, prioritize, and track defects</strong>, performance regressions, and alignment failures using Jira, collaborating closely with Applied AI Engineers to analyze root causes and validate fixes</li> <li><strong>Evaluate new technologies and patterns</strong> (e.g., new tools for Agent Evals), run proofs-of-concept, and influence the platform roadmap</li> <li><strong>Partner with policy and legal teams</strong> to ensure agent evaluations systematically cover regulatory compliance, fairness objectives, and emerging AI governance requirements</li> <li><strong>Provide technical leadership to engineers</strong>, including solutioning, design reviews, code reviews, pair programming, and mentoring of junior and contract engineers</li> <li><strong>Collaborate cross-functionally </strong>— work closely with ML engineers, product teams, platform engineers, and security to embed eval-driven quality gates into the agent development lifecycle</li> </ul> <h2><strong>Required Qualifications</strong></h2> <h3><strong>Education & Experience</strong></h3> <ul> <li>Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience</li> <li><strong>8+ years of overall software Test engineering experience</strong>, with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or Large Language Models (LLMs)</li> <li><strong>2+ years working with ML/AI systems, building</strong>, testing, or evaluating models in production environments, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agenti