LLMOps / Deployment¶
Last reviewed: October 2026
What's changed recently
- Serving engines consolidated. Hugging Face put TGI into maintenance mode and now recommends vLLM and SGLang (plus llama.cpp or MLX for local use). New self-hosted stacks should start there.
- Kubernetes-native distributed inference (prefill/decode disaggregation, KV-cache-aware routing via the Gateway API Inference Extension and llm-d-style schedulers) is how large self-hosted deployments scale now.
- Managed APIs added cost and latency tiers (batch, flex, priority) and cached-input pricing, which changes the managed vs self-host math.
- Agent operations got managed tooling. Online evaluators and trace-based monitoring now ship in platforms (for example AgentCore Evaluations and Snowflake Cortex Agent evaluations, both GA in March 2026).
LLMOps is MLOps for LLM apps: serving, scaling, cost control, caching, versioning, and lifecycle of models and prompts in production. It's what turns a working prototype into a reliable, affordable service.
Serving options¶
flowchart TB
APP[Your app / API] --> CHOICE{Serving path}
CHOICE -->|managed| API[Hosted API: Bedrock / OpenAI / Azure]
CHOICE -->|self-host| SRV[Inference server: vLLM / SGLang]
SRV --> GPU[(GPUs)]
API --> APP2[Response]
SRV --> APP2
| Path | Pros | Cons |
|---|---|---|
| Managed API (Bedrock, OpenAI, Azure) | No infra, fast to ship, scales | Per-token cost, less control, data-egress concerns |
| Self-hosted (vLLM, SGLang; Ollama/llama.cpp for local) | Control, data stays in, fixed cost | You manage GPUs, scaling, ops |
Performance & cost levers¶
- Caching — cache identical prompts/responses; prompt caching reuses a fixed prefix (system prompt, docs) across calls to cut cost/latency.
- KV cache — self-hosted servers reuse attention state across tokens (vLLM's paged KV cache boosts throughput).
- Batching — group concurrent requests for GPU efficiency.
- Quantization — 8-bit/4-bit weights shrink memory and speed inference with small quality loss.
- Model right-sizing — use the smallest model that passes eval; route easy queries to cheap models, hard ones to strong models (model routing).
- Streaming — stream tokens for perceived latency.
- Token budgets — cap input/output; trim context (see Context Engineering).
Lifecycle & governance¶
- Prompt versioning — treat prompts as code: version, review, test, roll back.
- Model versioning — pin model versions; test before upgrading (models drift).
- CI/CD with evals — run the eval suite on every prompt/model change (see Observability).
- Canary / A-B rollout — ship changes to a slice first.
- Guardrails + monitoring — in the request path and on metrics (cost, latency, error rate, quality).
- Secrets & data — keys in a secrets manager; know where data flows (governance, PII, region).
Reliability patterns¶
- Timeouts + retries on model calls; fallback to a smaller model or cached answer on failure.
- Rate limiting per user/tenant.
- Graceful degradation — return a useful partial/extractive answer if the LLM is down (like this site's own agent).
Interview deep dive¶
60-second talking points¶
- "Managed API to ship fast; self-host (vLLM/SGLang) for control and fixed cost."
- "Biggest cost levers: caching, right-sizing/routing, and token budgets."
- "Treat prompts and model versions as code, gated by an eval suite."
Scenario & system-design questions¶
Your LLM feature's bill is too high. How do you cut cost without wrecking quality?
Route easy queries to a cheaper model, reserve the strong model for hard ones (model routing); enable prompt caching for the fixed prefix; trim context with budgets; cache repeated queries; and cap max tokens. Measure quality on the eval set so cuts don't regress accuracy.
Design deployment for a customer-facing RAG chatbot.
App/API (FastAPI) → retrieval → managed model (Bedrock) or self-hosted vLLM/SGLang → guardrails → streaming response. Add caching, timeouts+retries with a fallback model, rate limiting, tracing/monitoring, prompt/model versioning, and CI evals. Canary new versions.
When would you self-host a model instead of using a managed API?
Data must stay in-house (compliance), you need a custom/fine-tuned model, want fixed cost at high steady volume, or need latency/control a hosted API can't give. Trade-off: you own GPU ops and scaling.
How do you safely upgrade the underlying model version?
Pin versions; run the eval suite on the new version; canary to a slice of traffic; monitor quality/cost/latency; roll back if metrics regress. Models drift, so never auto-upgrade blindly.
Pitfalls interviewers probe¶
- No caching / no token budgets (runaway cost).
- One giant model for every query (no routing).
- Prompts/models unversioned and untested.
- Auto-upgrading model versions without eval.
- No fallback when the model API fails.
Rapid-fire¶
| Q | A |
|---|---|
| Managed vs self-host? | Speed/no-ops vs control/fixed-cost/data-in |
| Self-host servers? | vLLM, SGLang (TGI is in maintenance mode); Ollama/llama.cpp locally |
| Prompt caching? | Reuse a fixed prefix across calls to cut cost/latency |
| Model routing? | Cheap model for easy queries, strong for hard |
| Safe model upgrade? | Pin, eval, canary, monitor, roll back |
| Quantization? | Lower-bit weights → less memory, faster, small quality loss |
Production AI CI/CD pipeline¶
Shipping AI safely means the pipeline gates on quality and safety, not just "the code compiles." A functional test suite passing is not enough for an AI system, an unchanged codebase can still regress because a prompt or model changed.
Kiro and Jenkins — how they fit together
In this workflow, Kiro is used for AI-assisted development and automation (writing code, scaffolding, refactors, docs). Jenkins remains the production CI/CD system that builds, tests, evaluates, scans, and deploys. Kiro speeds up authoring; Jenkins is still the gate to production. Kiro does not replace Jenkins.
flowchart TB
DEV[Developer] --> KIRO[Kiro-assisted development<br/>author / refactor / automate]
KIRO --> GIT[Git commit / PR]
GIT --> JENK[Jenkins pipeline]
JENK --> UT[Unit + integration tests]
UT --> EVAL[AI evaluation<br/>golden + regression sets]
EVAL --> SEC[Security scanning<br/>deps, secrets, prompt-injection suite]
SEC --> DEP[Deploy<br/>canary / blue-green]
DEP --> OBS[Observability<br/>quality, cost, latency, safety]
OBS --> RB{Regression / SLO breach?}
RB -- yes --> ROLL[Rollback]
RB -- no --> DONE[Promote]
What each AI-specific gate does:
- Golden-dataset evaluation — a curated set of representative inputs with known-good outputs/rubrics. The change must meet a quality bar (correctness, faithfulness, format validity) before it proceeds.
- Regression evaluation — compares the new prompt/model against the current production version on the same set; blocks a drop even if absolute scores look fine.
- Security scanning — dependency/secret scans plus an AI layer: run a prompt-injection / jailbreak test suite and check output guardrails still hold.
- Canary + rollback — ship to a traffic slice, watch quality/cost/latency/ safety, auto-roll-back on regression. Deploy is decoupled from release via flags.
What's versioned and gated (treat all as code):
| Artifact | Versioned | Gated by |
|---|---|---|
| Application code | Git | Unit/integration tests |
| Prompts | Prompt registry / Git | Golden + regression eval |
| Model version | Pinned config | Eval before upgrade (models drift) |
| Datasets / eval sets | Dataset versioning | Reviewed; drives the gates |
| Retrieval index config | Config as code | Retrieval eval (recall@k) |
AgentOps — operating agents in production¶
Agents add operational surface beyond a single model call: multi-step loops, tools, memory, and non-determinism. AgentOps extends LLMOps with agent-specific lifecycle and telemetry.
Beyond LLMOps, you also manage:
- Tool registry + versioning — tools (and MCP servers) are dependencies; pin, review on change, and eval routing accuracy after changes.
- Trajectory evaluation — score not just the final answer but the path (were the steps/tool calls sensible and efficient?). See Observability & Eval.
- Loop / cost guards as first-class ops — step caps, token/cost budgets, and repeated-action detection are runtime controls you monitor and alert on.
- Human-in-the-loop queues — approvals for gated actions become an operational workflow (latency, backlog, audit).
- Replay + tracing — durable, checkpointed state lets you replay a bad run to find which step failed (see Agent Principles).
Agent-specific signals to monitor: steps-per-task, tool-failure rate, retry count, loop-detection hits, cost-per-task, approval latency, and task-success rate over time (drift).
How is deploying an agent different from deploying a single-call LLM feature?
An agent has a loop, tools, and memory, so more can go wrong and it's non-deterministic. You add: tool/MCP versioning + review, trajectory eval (not just final-answer), runtime loop/cost guards you monitor, HITL approval queues, and step-level tracing for replay. CI must eval the whole trajectory and the routing, not just one prompt.
A prompt change passes all unit tests but the AI answers got worse. How does CI catch this?
Unit tests can't see quality regressions. The pipeline needs a golden-dataset eval (meet a quality bar) plus a regression eval (compare against the current prod version and block a drop). That's the AI-specific gate between "tests pass" and "deploy."
Where does Kiro fit vs Jenkins in your delivery pipeline?
Kiro is for AI-assisted development — authoring, refactoring, automating work in the editor. Jenkins is the production CI/CD that runs tests, AI evaluation, security scanning, and controls the deploy. Kiro accelerates how code gets written; Jenkins is still the gate that decides what reaches prod.
Related
Observability & Eval · Cost Optimization · Agent Principles · AI Security · Practice: DevOps Interview Q&A · AI Engineer Interview Q&A