<p>At Qualia, we've built the leading B2B real estate technology that transforms the home buying and selling experience into a simple, secure, and enjoyable process. Our SMB and Enterprise products bring together users from across the real estate ecosystem—homebuyers and sellers, lenders, title and escrow agents, and real estate agents—onto a single shared digital closing platform, providing greater clarity and transparency to real estate transactions. Today, through our business customers across the country, millions of consumers use Qualia to close on homes every year.</p> <h3><strong>WHAT YOU'LL WORK ON</strong></h3> <p>We are hiring pioneering and collaborative engineers to build our next generation of intelligent products. Our quality engineers are conscientious hackers who thrive on collaboration and are adept at driving large-scale projects in the fast-changing domain of artificial intelligence. You will help build the automation, tooling, observability, and quality systems that let us ship complex, AI-driven products with confidence, bringing rigor and trust to a data-rich industry.</p> <p>As a Senior AI/Agentic Quality Infrastructure SDET, you're a software engineer first. You build test infrastructure, agentic validation systems, AI/ML evaluation harnesses, observability platforms, and engineering tooling—not manual test cases or collections of automation scripts. You will help evolve our quality architecture from primarily deterministic automation toward a combination of traditional automation and agentic validation, building systems where AI agents can understand application state, execute workflows, use engineering tools, validate outcomes, investigate failures, and provide evidence for their conclusions.</p> <p>You can take a hard, loosely defined problem and solve it. You're trusted to make technical decisions on frameworks, architecture, evaluation strategy, observability, and validation across every layer of the testing pyramid. You work closely with Engineering and Quality leadership, understand the systems you're testing at a deep technical level, and mentor engineers on building reliable tests, tooling, infrastructure, and AI-assisted validation systems.</p> <h3><strong>RESPONSIBILITIES</strong></h3> <ul> <li>Build and maintain test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms as production-grade software rather than scripts</li> <li>Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows, reasoning about application state rather than relying solely on predefined test scripts</li> <li>Build AI/ML evaluation frameworks for validating non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, and human-in-the-loop review, defining measurable quality signals such as correctness, reliability, latency, cost, and hallucination/error rates</li> <li>Build observability and telemetry into our automation and agentic infrastructure—structured logs, metrics, distributed tracing, and execution timelines—so engineers can understand why an automated or agentic workflow succeeded or failed</li> <li>Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis</li> <li>Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale</li> <li>Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence</li> <li>Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering</li> </ul> <h3><strong>YOUR BACKGROUND THAT LIKELY MAKES YOU A MATCH</strong></h3> <ul> <li>5+ years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems—not primarily writing test cases or automation scripts</li> <li>Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch</li> <li>Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations</li> <li>Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing</li> <li>Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows</li> <li>Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non-deterministic systems</li> <li>Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, or agentic layer&l