Skip to content

Cost Optimization

Last reviewed: October 2026

What's changed recently

  • More price tiers. Most providers now offer batch APIs (often around half the on-demand price), discounted cached input tokens, and latency tiers. Bedrock added Priority, Standard, and Flex tiers in November 2025.
  • Reasoning tokens are a major cost line. Thinking or reasoning models bill for hidden reasoning output, so reasoning-effort or thinking-budget settings are now real cost controls.
  • Agents multiply tokens. Per-token prices for a given capability keep falling, but agent loops, tool schemas, and long contexts raise volume, so cost per successful task is the metric that matters.

LLM bills scale with tokens × calls × model price. This page is the practical playbook for cutting cost without wrecking quality. See also LLMOps.

Where the money goes

flowchart LR
    COST[Cost] --> T[Tokens in + out]
    COST --> C[Number of calls]
    COST --> M[Model price tier]
    T --> CTX[Context size / RAG chunks]
    C --> AGENT[Agent iterations]
    M --> ROUTE[Model choice]

The levers (biggest first)

Lever How Typical savings
Model routing Cheap model for easy queries, strong for hard Large
Prompt caching Reuse a fixed prefix (system + docs) across calls Large for repeated prefixes
Right-sizing Smallest model that passes eval Large
Token budgeting Cap input (trim context/RAG) and output (max_tokens) Medium
Response caching Cache identical/similar queries Medium (repetitive traffic)
Fewer agent steps Iteration caps, better tools/plans Medium
Batch / flex tiers Send latency-tolerant work to provider batch APIs or flex tiers Large for offline jobs
Reasoning effort Lower thinking budget / effort on easy tasks Medium–large on reasoning models
Batching Group requests (self-host) Medium (throughput)
Shorter prompts Trim verbose system prompts / few-shot Small–medium

A cost-aware request

flowchart TB
    Q[Query] --> CACHE{Cached?}
    CACHE -- yes --> RET1[Return cached]
    CACHE -- no --> ROUTE{Hard?}
    ROUTE -- easy --> SMALL[Small model]
    ROUTE -- hard --> BIG[Strong model]
    SMALL & BIG --> BUD[Budget context + output]
    BUD --> CALL[Call w/ prompt caching]
    CALL --> STORE[Cache result]

Guardrails so cuts don't hurt quality

  • Measure on an eval set — every cost change is scored for quality regression, not guessed.
  • Watch the metric that matters — cost per successful task, not per call.
  • Monitor cost per user/tenant/day; alert on spikes.

Common wins

  • Move classification/routing/extraction to a small model.
  • Turn on prompt caching when the system prompt or retrieved docs repeat.
  • Trim RAG — retrieve wide, rerank, keep only the top few chunks.
  • Cap agent iterations — most runaway cost is agent loops.

Interview deep dive

Talking points

  • "Cost = tokens × calls × price — attack all three."
  • "Biggest levers: routing, caching, right-sizing."
  • "Measure cost per successful task, gated by an eval set."

Scenario questions

Your GenAI feature's monthly bill doubled. Walk through cutting it.

Break down by tokens/calls/model. Route easy queries to a cheaper model; enable prompt caching for the fixed prefix; trim context (fewer, reranked chunks; max_tokens); cache repeated queries; cap agent iterations. Re-run the eval set so quality holds.

How do you decide if a cheaper model is 'good enough'?

Evaluate it on a labeled set for the actual task; if it passes the quality bar (faithfulness/accuracy), route eligible traffic to it and keep the strong model for the hard slice.

Rapid-fire

Q A
Cost drivers? Tokens (in+out), call count, model price
Biggest lever? Model routing + right-sizing
Prompt caching? Reuse a fixed prefix to cut cost/latency
Right metric? Cost per successful task
Runaway agent cost fix? Iteration + cost caps
Offline workload? Batch API / flex tier (often ~50% cheaper)

How interviewers probe this

Finance wants a 40% cut with no quality loss. How do you plan it?

A strong answer starts with attribution: cost by feature, tenant, model, and token type (input, cached input, output, reasoning). Then go after the biggest bucket with the matching lever: routing or right-sizing, prompt caching for repeated prefixes, batch or flex for offline work, trimming context, and capping agent loops. Each change ships behind an eval gate and a canary, and progress is reported as cost per successful task.

How do you design a model router, and how do you know it's working?

Use a cheap classifier (rules, a small model, or confidence from a first pass) to decide the tier, escalate on low confidence or failed validation, and log each decision. Check it with offline evals on a labeled set, then online metrics: the share of traffic handled by the cheap tier, the escalation rate, and quality per tier. Misrouting hard queries to a weak model is the failure to watch.

Prompt caching is on, but the hit rate is low. Why?

The prefix isn't stable. Common causes: timestamps or user IDs early in the system prompt, tool definitions in a non-deterministic order, retrieved chunks placed before stable instructions, or prefixes below the provider's minimum cacheable length. Fix the layout so the static parts come first and byte-identical.

How do you stop one tenant from blowing the monthly budget?

Per-tenant token and cost budgets enforced at the gateway, rate limits, request-size caps, per-tenant cost dashboards with alerts, and graceful degradation (cheaper model or queued processing) instead of a hard failure when a soft limit is reached.


See also: AI economics in 2026: who gets paid along the chain from chips to apps, and how interviewers ask about it.