Senior LLMOps Engineer

About The Role

Kong’s API Management suite including the Kong Enterprise AI Gateway and Konnect support multiple AI capabilities for customers.

The AI Gateway offers a unified API for managing multiple Large Language Models (LLMs), allowing users to switch between different AI providers seamlessly. The AI Gateway also allows a variety of AI policies to be enforced such as semantic caching, advanced routing and load balancing, promt guards and Personally Identifiable Information (PII) sanitization. The Konnect SaaS platform exposes these same AI capabilities via Cloud Gateways and provides customers a variety of AI-driven insights into the operations of a customer’s API surface.

We are building out a new platform team - LLMOps - in our Bangalore office to help all the other engineering teams in Kong work with Large Language Models (LLMs) and build new features into our product portfolio.

Be the founding member of the LLMOps team

Provide an LLM SaaS platform for all the engineering teams at Kong, giving them access to a variety of LLMs based on their needs

Manage model lifecycle with high reliability and uptime SLOs (at least 99.99%)

Fine tune the platform to provide AI security and high throughput for LLM queries

Own the end-to-end deployment architecture of the LLM SaaS platform

Play a pivotal role in shaping the technical direction for AI innovation in Kong

Collaborate closely with product leadership to define and refine strategy, roadmap, and objectives for the LLM SaaS platforms, ensuring alignment between engineering efforts and business goals.

Assist in the hiring process and help grow and mentor the team

Bachelor's or Master's degree in Computer Science or a related field

5+ years of experience in building and operating highly reliable SaaS/PaaS systems

Prior ownership of uptime SLOs for a platform operating at 99.99% or higher

Hands-on experience with at least one of the major cloud platforms (AWS, Azure, or GCP)

Strong experience with AI platforms in public clouds (like Bedrock, Vertex AI, etc.).

Familiarity with observability tools such as Datadog, Prometheus, Grafana, Victoria Metrics, Loki, or similar technologies.

Expertise in designing and developing highly scalable distributed systems

Experience managing incidents and communicating effectively under high-pressure situations

Strong verbal and written communication skills to effectively collaborate across teams