Monitoring & Observability¶
Last reviewed: October 2026
A GenAI service fails in ways a normal service doesn't. It can return HTTP 200 with a confident, wrong answer. It can stay fast while a prompt change doubles token spend. A retrieval index can go stale and quietly lower answer quality. An agent can loop through tools for four minutes before timing out. Standard RED metrics (rate, errors, duration) are necessary but they don't catch most of these.
Production monitoring for LLM systems therefore has five signal families: operational, cost, quality, safety and change. They share one tracing backbone. This page covers the run-time side. For offline evaluation design see Observability & Eval.
Reference design¶
flowchart LR
subgraph App
GW[AI gateway] --> AG[Agent runtime]
AG --> RET[Retrieval]
AG --> LLM[Model calls]
AG --> TL[Tool calls]
end
GW & AG & RET & LLM & TL -->|OTLP spans, metrics, events| COL[OTel Collector<br/>redact, sample, route]
COL --> TR[(Trace store)]
COL --> MET[(Metrics / TSDB)]
COL --> AUD[(Audit pipeline)]
TR --> EVAL[Online evaluators<br/>LLM-as-judge on a sample]
EVAL --> MET
FB[User feedback] --> MET
MET --> DASH[Dashboards + SLO alerts]
TR --> DS[Failure cases -> eval dataset]
Points to defend in a design review:
- One trace per user turn, with child spans for retrieval, every model call, every tool call and guardrail checks. Without this, cost and latency can't be attributed to a step.
- The Collector is the policy point. It redacts content, applies tail-based sampling (keep all errors, slow traces and guardrail hits; sample the rest), and routes complete security events to the audit pipeline without sampling.
- Quality is measured online, on a sample. Run cheap heuristics on all traffic, and run LLM-as-judge or rubric graders on a sample. Feed bad cases back into the offline eval set.
OpenTelemetry GenAI semantic conventions¶
OpenTelemetry defines GenAI conventions for spans, metrics and events. They have since moved out of the main semantic-conventions docs into a dedicated repository, open-telemetry/semantic-conventions-genai. Most of the GenAI conventions are still pre-stable, and names have changed between releases. For example, gen_ai.system was superseded by gen_ai.provider.name. Pin your instrumentation versions and check the spec before you build dashboards on attribute names.
| Kind | Name | Use |
|---|---|---|
| Span attribute | gen_ai.operation.name |
chat, embeddings, execute_tool, invoke_agent, create_agent, … |
| Span attribute | gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model |
Which provider and model; the response model catches silent aliasing |
| Span attribute | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |
Cost attribution per span |
| Span attribute | gen_ai.response.finish_reasons |
Spot truncation (length) and filter stops |
| Span attribute | gen_ai.conversation.id, gen_ai.agent.name, gen_ai.tool.name |
Group by session, agent and tool |
| Metric | gen_ai.client.token.usage |
Histogram of tokens, split by gen_ai.token.type |
| Metric | gen_ai.client.operation.duration |
Client-side latency histogram |
| Metric | gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token |
Streaming latency on self-hosted serving |
Message content (prompts, completions, tool arguments) is opt-in. Instrumentations typically gate it behind a flag, for example OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT in the OpenTelemetry Python GenAI instrumentations. Keep it off in production unless content goes to a governed store.
Collector: strip content before it reaches shared backends¶
processors:
attributes/strip_genai_content:
actions:
- key: gen_ai.input.messages
action: delete
- key: gen_ai.output.messages
action: delete
- key: gen_ai.system_instructions
action: delete
tail_sampling:
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow
type: latency
latency: { threshold_ms: 15000 }
- name: baseline
type: probabilistic
probabilistic: { sampling_percentage: 10 }
service:
pipelines:
traces/shared:
receivers: [otlp]
processors: [attributes/strip_genai_content, tail_sampling, batch]
exporters: [otlp/apm]
Add your own platform attributes alongside the standard ones: tenant.id, app.name, prompt.version, index.snapshot, guardrail.action, cost.usd.
What to measure and alert on¶
| Family | Metrics | Alert when |
|---|---|---|
| Operational | TTFT p50/p95, total latency, error rate by error.type, provider throttling (HTTP 429), timeouts, agent steps per turn |
SLO burn rate (multi-window); throttling sustained; steps per turn p95 jumps |
| Cost | Tokens and $ per request, user, tenant, app; cache hit rate; cost per successful task | Daily spend above forecast; per-user anomaly; cache hit rate drops after a deploy |
| Quality | Groundedness or faithfulness score (sampled), citation coverage, thumbs-down rate, retrieval empty-result rate, task success rate for agents | Week-over-week drop beyond threshold; empty retrievals spike (often an index or filter bug) |
| Safety | Guardrail interventions by category, prompt-attack detections, PII redactions, tool-policy denials, approval rejections | Sudden surge (attack or bad deploy); a new category appears |
| Change | Model ID served, prompt version, index snapshot age, guardrail config version | Unplanned change; index older than freshness SLO |
Two practical rules:
- Page on symptoms users feel (latency SLO, error rate, availability). Quality and cost drift go to tickets or daytime alerts unless they are severe. LLM-judge scores are noisy, and paging on them burns out the on-call.
- Normalize cost by outcome. "$0.04 per request" is less useful than "$0.31 per resolved ticket". A cheaper prompt that needs three retries isn't cheaper.
Example SLOs¶
| SLO | Target (example) | Measured from |
|---|---|---|
| Chat availability | 99.9% of turns return a non-error response | Gateway spans |
| Interactive latency | p95 TTFT < 2 s | invoke_agent / chat spans |
| Agent completion | 95% of agent tasks finish under the step budget | Agent spans |
| Index freshness | 99% of source changes searchable within 1 h | Ingestion pipeline |
The targets are illustrative. Set yours from measured baselines and product needs.
Failure modes and anti-patterns¶
What teams get wrong
- Dashboards with averages only. LLM latency is long-tailed; use p95 and p99 and split by model and prompt version.
- No
response.model. A provider alias moves to a new model version and quality shifts with no visible change. - Head-based sampling at 1%, which throws away almost every rare failure you care about.
- Content capture on by default in a shared APM tool. The observability vendor now holds your customers' PII.
- Quality measured only offline. The eval set passes while production traffic has drifted to topics it doesn't cover.
- No canary comparison. Prompt and model changes go to 100% at once, so you can't separate a regression from noise.
- Agent loops are invisible, because each model call succeeds. Track steps, repeated tool calls and wall-clock time per task.
How interviewers probe this¶
Thumbs-down rate doubled overnight. Latency and errors are flat. How do you debug?
Check change signals first: model served, prompt version, index snapshot, guardrail config. Then segment by tenant, topic and channel. Pull sampled traces of thumbs-down turns and compare retrieval results and groundedness scores with a baseline week. Typical causes are a stale or partially rebuilt index, a filter bug returning empty context, or a provider model update.
How do you attribute LLM cost to teams and customers?
Tag every span with tenant, app and cost center at the gateway. Compute cost from token usage and a versioned price table, including cached-token and batch discounts. Reconcile monthly against the provider invoice. Cover showback dashboards, budgets with soft and hard limits, and alerts on anomalies.
What would you sample, and what must never be sampled?
Tail-sample traces: keep errors, slow traces, guardrail hits and negative feedback, and sample the rest. Never sample audit events, policy denials or security detections. Those go through a separate unsampled path.
How do you monitor quality without labels?
Use proxy signals: groundedness and citation checks, LLM-as-judge on a sample against a rubric, retrieval health (empty or low-score results), refusal rate, explicit and implicit feedback (rephrasing, abandonment). Calibrate judges against periodic human review.
How do you roll out a new model safely?
Run an offline eval gate first, then shadow traffic or a canary on a small slice. Compare quality, latency, cost and safety metrics with statistical thresholds. Promote in stages, keep automatic rollback, and pin model versions so the provider can't change them under you.
Checklist¶
- One trace per turn with spans for retrieval, model, tool and guardrail steps
- OTel GenAI attributes plus tenant, app, prompt version and index snapshot
- Content capture off in shared backends; redaction in the Collector
- Tail-based sampling; audit and security events unsampled
- SLOs for availability, TTFT and agent completion, with burn-rate alerts
- Cost per outcome, per tenant, with budgets and anomaly alerts
- Online quality sampling with calibrated judges, and failures fed into the eval set
- Canary comparison for every model, prompt or index change
Further reading¶
- OpenTelemetry GenAI semantic conventions repository
- OpenTelemetry Collector documentation
- Google SRE Workbook: Alerting on SLOs
- AWS Well-Architected Generative AI Lens