Splunk Infrastructure Engineer

<div class="content-intro"><p><strong>Secure Every Identity, from AI to Human<br><br></strong>Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence.<br><br>This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk.</p></div><div class="SingleJob-content"> <h3><strong>Position Overview:</strong></h3> <p>We are seeking a highly technical <strong>Splunk Infrastructure</strong><strong> Engineer</strong> with a specialty in Splunk to own and evolve our Splunk ecosystem. In this role, you will move beyond simple monitoring to delivering a world class, comprehensive, scalable Observability Platform that enables our SRE teams and business partners. </p> <p><strong>Key Responsibilities</strong></p> <ul> <li>Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services</li> <li>Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."</li> <li>Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.</li> </ul> <p><strong>Required Skills & Experience (The Essentials)</strong></p> <p><strong>Log Management: Minimum 5+ Experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization. </strong><strong>Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.</strong></p> <ul> <li><strong>Problem Solving:</strong> A data-driven approach to debugging complex, cross-service performance bottlenecks.</li> </ul> <p><strong>Bonus Skills (The "Nice-to-Haves")</strong></p> <ul> <li><strong>Telemetry Standards: </strong>Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.</li> <li><strong>Charge-back app:</strong> Experience in implementing Splunk charge-back app for usage reporting </li> </ul> <p><strong>Cloud Platforms:</strong> Experience managing observability native tools within AWS or GCP.</p> <p><strong>Additional requirements:</strong></p> <ul> <li>This position requires the ability to access federal environments and/or have access to protected federal data.  As a condition of employment for this position, the successful candidate must be able to submit documentation establishing U.S. Person status (e.g. a U.S. Citizen, National, Lawful Permanent Resident, Refugee, or Asylee. 22 CFR 120.15) upon hire.</li> <li>This person must attend in person onboarding in our San Francisco/Chicago office the first week of employment. </li> </ul> <p>#LI-MM</p> <p>#LI-Hybrid<br>P14596_3372199</p> </div> <p> </p><div class="content-pay-transparency"><div class="pay-input"><div class="title">The annual base salary range for this position for candidates located in the San Francisco Bay area is between: </div><div class="pay-range"><span>$194,000</span><span class="divider">—</span><span>$267,000 USD</span></div></div><div class="pay-input"><div class="description"><p><span style="font-weight: 400;">Below is the annual base salary range for candidates located in California (excluding San Francisco Bay Area), Colorado, Illinois, New York and Washington. Your actual base salary will depend on factors such as your skills, qualifications, experience, and work location. In addition, Okta offers equity (where applicable), bonus, and benefits, including health, dental and vision insurance, 401(k), flexible spending account, and paid leave (including PTO and parental leave) in accordance with our applicable plans and policies. To learn more about our Total Rewards program please visit: </span><a href="https://rewards.okta.com/us"><span style="font-weight: 400;">https://rewards.okta.com/us</span></a><span style="font-weight: 400;">.   <br></span></p></div><div class="title">The annual base salary range for this position for candidates located in California (excluding San Francisco Bay Area), Colorado, Illinois, New York, and Washington is between:</div><div class="pay-range"><span>$174,000</span><span class="divider">—</span><span>$238,000 USD</span></div></div></div><div class="content-conclusion"><p><strong>The Okta Experience</strong></p> <ul> <li><a href="http://rewards.okta.com">Supporting Your Well-Being</a> </li> <li><a href="https://www.okta.com/company/okta-for-good/">Driving Social Impact </a></li> <li><a href="https://www.okta.com/company/talent-connection-community/">Developing Talent and Fostering Connection + Community</a></li> </ul> <p>We are intentional about connection. Our global community, spanning over 20 offices worldwide, is unite