AI Operations Specialist – LLM, Prompt Engineering & Conversation Systems | Remote Position Type: Full-Time, Remote Working Hours: U.S. Business Hours Industry: Artificial Intelligence / SaaS / Telehealth Technology Department: AI Operations / Conversational AI Focus: Production LLM Systems, AI Conversation Optimization & Conversion Performance About the Role At Pavago, one of our clients is hiring a highly skilled AI Operations Specialist to manage, monitor, troubleshoot, and continuously optimize production AI conversation systems. Our client operates an AI-powered SMS lead recovery platform serving telehealth brands, processing millions of messages every month. These AI systems communicate directly with customers, making conversation quality, reliability, and conversion performance critical to business success. This is a hands-on, execution-focused role for someone with proven experience operating Large Language Model (LLM) systems in production, rather than simply experimenting with AI tools. You’ll be responsible for onboarding new clients, configuring AI conversation workflows, managing knowledge bases and integrations, monitoring live customer interactions, identifying AI failure modes, optimizing prompts, and measuring the impact of changes on conversion performance. The ideal candidate combines technical AI knowledge, prompt engineering expertise, analytical thinking, and operational ownership with the ability to work independently in a fast-moving environment. You’ll collaborate directly with the founder to ensure AI systems consistently deliver accurate, effective, and conversion-focused customer conversations. What You’ll Own Client Onboarding & AI System Configuration • Onboard new clients onto the AI-powered conversation platform. • Configure client-specific AI conversation systems and workflows. • Set up and troubleshoot webhook integrations, APIs, and knowledge bases. • Configure SMS campaigns and conversation workflows. • Ensure AI systems are aligned with client requirements and objectives. • Test conversation flows and integrations before deployment. • Identify and resolve configuration issues during onboarding. • Verify that new client systems are functioning correctly before going live. • Maintain accurate documentation of client configurations and setup processes. Production AI Operations & Monitoring • Monitor live AI-generated conversations and customer interactions. • Review conversation transcripts to identify errors, inconsistencies, and missed conversion opportunities. • Diagnose AI behavior issues and determine their underlying causes. • Identify patterns affecting conversation quality, accuracy, and performance. • Detect unexpected AI responses, workflow failures, and system inconsistencies. • Investigate issues involving conversation logic, prompts, knowledge bases, or integrations. • Escalate technical issues when necessary and follow through until resolution. • Maintain consistent quality standards across customer-facing AI systems. • Proactively identify opportunities to improve production reliability and conversation effectiveness. Prompt Engineering & LLM Optimization • Write, edit, and optimize system prompts and customer-facing AI messaging instructions. • Improve AI conversation logic based on real customer interactions. • Identify weaknesses in existing prompts and system behavior. • Adjust instructions to improve response quality, accuracy, and consistency. • Test prompt changes systematically before deployment. • Evaluate AI behavior before and after implementing changes. • Document prompt updates, testing results, and system modifications. • Continuously refine AI-generated responses to improve customer engagement. • Ensure prompt changes support both conversation quality and conversion objectives. AI Conversation Analysis & Quality Assurance • Review large volumes of AI-generated conversation transcripts. • Identify recurring response errors and behavioral patterns. • Analyze missed opportunities within customer conversations. • Evaluate whether AI responses appropriately address customer needs. • Investigate inconsistencies between expected and actual AI behavior. • Identify opportunities to improve conversation flows and response handling. • Conduct structured testing of prompts and conversation workflows. • Maintain quality standards across deployed AI systems. Conversion Performance & Data Analysis • Analyze conversation performance and customer interaction data. • Measure the impact of AI system changes on lead conversion and engagement. • Identify patterns that influence customer responses and conversion outcomes. • Track performance before and after prompt or workflow adjustments. • Evaluate whether changes deliver measurable improvements. • Use conversation data to prioritize optimization opportunities. • Communicate performance findings and recommendations to leadership. • Support data-driven decisions regarding AI conversation strategies. Founder Collaboration & Operational Ownership • Work directly with the founder on daily AI operations and improvement initiatives. • Provide clear updates on system performance, issues, and optimization opportunities. • Identify operational problems and take ownership of resolving them. • Recommend improvements based on production data and customer interactions. • Prioritize urgent issues affecting customer-facing AI systems. • Coordinate technical escalations when necessary. • Document findings, system changes, and testing outcomes. • Move quickly while maintaining high standards for AI reliability and accuracy. Requirements Must-Have Qualifications • Proven experience operating LLM or AI systems in production — essential. • Hands-on experience monitoring, troubleshooting, and improving live AI-generated outputs. • Strong understanding of Large Language Models (LLMs) and their behavior in real-world applications. • Practical experience with prompt engineering and system prompt optimization. • Experi