Skip to content

Reliability & Distributed Systems

Last reviewed: October 2026

An AI system is a distributed system with an unusually flaky dependency (the model/tool layer) bolted on. This page covers the reliability patterns every senior engineer must apply, then the AI-specific failure modes and how each pattern contains them.

The core assumption

Every remote call will fail, time out, or return garbage eventually — model providers, tools, vector DBs, MCP servers. Reliability is designing so that a dependency failure degrades gracefully instead of cascading.


The reliability toolkit

Pattern Problem it solves AI-system use
Timeout A slow call blocks forever Cap every model/tool/retrieval call
Retry + backoff + jitter Transient failures Retry model/tool on 429/5xx, spread the load
Circuit breaker A dead dependency wastes calls Stop hammering a down model provider; fail fast
Bulkhead One workload starves others Isolate the agent pool from the ingestion pool
Rate limiting Overload / cost abuse Per-user/tenant caps on requests + tokens
Idempotency Duplicate side effects on retry Keyed writes so a retried tool call doesn't double-act
Queue + DLQ Bursty load, poison messages Async work; dead-letter the un-processable
Backpressure Producers outrun consumers Bound queues; shed/slow load instead of OOM
Fallback / graceful degradation Dependency down Cheaper model, cached answer, or extractive reply

Retry with backoff + jitter (the one people get wrong)

Naive retries make an outage worse (thundering herd). Exponential backoff with jitter spreads retries so a recovering provider isn't re-flooded.

import random, time

def call_with_retry(fn, attempts=4, base=0.5, cap=8.0):
    for i in range(attempts):
        try:
            return fn()
        except Transient as e:          # only retry transient (429/5xx/timeout)
            if i == attempts - 1:
                raise
            sleep = min(cap, base * 2 ** i)      # exponential
            sleep = random.uniform(0, sleep)     # full jitter — key part
            time.sleep(sleep)
  • Only retry transient errors (429, 5xx, timeouts) — never a 400/validation error (it'll fail identically).
  • Jitter is not optional — without it, all clients retry in lockstep and re-DDoS the recovering service.
  • Cap attempts + total time so a retry storm can't blow your latency budget.

Circuit breaker

After N consecutive failures, "open" the circuit: fail fast (or go straight to fallback) for a cooldown, then let one trial request "half-open" test recovery. Stops you burning latency/cost calling a provider that's clearly down.

Idempotency (critical once agents take actions)

# A retried "create ticket" must not create two tickets.
def create_ticket(idempotency_key, payload):
    if store.exists(idempotency_key):        # already done → return prior result
        return store.get(idempotency_key)
    result = ticketing.create(payload)
    store.put(idempotency_key, result)
    return result

Any tool with a side effect needs an idempotency key so retries/replays are safe.


AI-specific failure scenarios

Each maps to patterns above. This table is a favorite interview probe.

Failure Symptom Containment
Model provider unavailable 5xx/timeout spike, errors Circuit breaker → fallback model or cached/degraded answer
Model provider rate-limited 429s Backoff+jitter, per-tenant rate limit, request queue
Tool/MCP server down Tool call errors Timeout + retry; agent proceeds with reduced tools or asks for human
Vector DB unavailable Retrieval empty/errors Fallback to keyword/cache; answer "can't retrieve now" — don't hallucinate
Malformed tool/model output Parse/schema failure Validate → one repair retry → fail loud; never eval it
Infinite agent loop Steps climb, cost spikes Step cap + repeated-action detection (bounded autonomy)
Context overflow Prompt exceeds window Compact/summarize; truncate oldest; retrieve less
Partial multi-tool failure 1 of N tools fails Decide: proceed with partial, retry the one, or abort — don't silently drop
Provider latency spike p95 doubles Timeout + fallback; parallelize independent calls; stream
Cost runaway Spend alarm Token/cost budget per task, cap loops, alert + kill switch

Worked scenario: one model provider goes down at peak

Symptom: error rate jumps, latency climbs as calls hang. Contain: timeouts stop the hangs; the circuit breaker opens after N failures so you fail fast instead of queueing; requests fall back to an alternate provider/region or a cheaper model; if none, return a cached or extractive answer with a "degraded" notice. Prevent recurrence: multi- provider config, health checks, and a load test that exercises the failover.


Distributed-systems fundamentals (the "why")

  • Everything is best-effort over an unreliable network — assume reordering, duplication, and partial failure.
  • At-least-once vs exactly-once — most delivery is at-least-once, so your sink must be idempotent to get effective exactly-once.
  • Eventual consistency — a freshly written record may not be visible immediately (e.g. a just-indexed doc). Design for it; don't assume read-your-write.
  • Backpressure over buffering — unbounded queues just move the OOM downstream; bound them and shed/slow load.
  • Bulkheads — isolate resource pools so one runaway workload (a stuck agent loop) can't starve the rest.

Interview deep dive

60-second talking points

  • "Every remote call fails eventually — timeout, retry with backoff+jitter, circuit-break, and fall back."
  • "Idempotency is what makes retries and agent actions safe."
  • "Degrade gracefully: a cached/extractive answer beats an error or a hang."
A model provider becomes unavailable in production. What happens to your system?

Timeouts stop calls from hanging; a circuit breaker opens after repeated failures so you fail fast; requests fall back to an alternate provider/ region or a cheaper model, or return a cached/extractive answer marked degraded. Backoff+jitter on retries avoids re-flooding it on recovery. Prevent: multi-provider config + a tested failover path.

Why is exponential backoff not enough without jitter?

Without jitter, every client backs off on the same schedule and retries in lockstep, re-DDoSing the recovering service (thundering herd). Full jitter randomizes each client's delay so load spreads out and the service can recover.

How do you make an agent's tool actions safe to retry?

Idempotency keys: each side-effecting tool call carries a key; the tool checks if that key was already processed and returns the prior result instead of acting twice. Combined with timeouts + bounded retries, a retried or replayed "create/refund/update" won't double-act.

Your agent occasionally hangs and burns cost. Which patterns apply?

Bounded autonomy (step cap + repeated-action detection) stops loops; timeouts stop a hanging tool/model call; a cost budget per task caps spend; bulkheads keep the stuck workload from starving others; a kill switch + alert lets ops intervene.

Retrieval (vector DB) is down. What should the agent do?

Fail gracefully, not falsely. Timeout the call; fall back to keyword search or a cache if available; otherwise answer "I can't retrieve the sources right now" rather than hallucinating from memory. Log it; it's a dependency incident.

Pitfalls interviewers probe

  • Retrying non-transient errors (400/validation) — wastes calls, same failure.
  • Exponential backoff without jitter (thundering herd).
  • No timeouts → hung calls exhaust the pool.
  • Side-effecting tools with no idempotency → duplicate actions on retry.
  • Unbounded queues instead of backpressure → OOM moved downstream.
  • Hallucinating when retrieval fails instead of degrading honestly.

Rapid-fire

Q A
Retry only when? Transient errors (429/5xx/timeout)
Why jitter? Prevents synchronized retry storms (thundering herd)
Circuit breaker? Fail fast after N failures; test recovery via half-open
Idempotency? Retried side-effect doesn't double-act (keyed)
Bulkhead? Isolate resource pools so one can't starve others
Backpressure? Bound queues; shed/slow load vs buffering to OOM
Provider down? Circuit-break → fallback model / cached answer
Retrieval down? Degrade honestly; never hallucinate