FastAPI¶
Last reviewed: October 2026
What's changed recently
- The FastAPI CLI (
fastapi dev,fastapi run) is the documented way to run apps, with--workers Nfor multiple processes. The docs no longer recommend Gunicorn as a process manager, and on Kubernetes they suggest one Uvicorn process per container. - Native Server-Sent Events. Recent releases include
fastapi.sse.EventSourceResponseandServerSentEvent, so you don't need a third-party package or hand-formatteddata:lines to stream LLM tokens. - Write new apps against Pydantic v2. Pydantic v1-style models are a legacy path; check your FastAPI version's support notes before relying on them.
FastAPI is a modern, high-performance Python web framework for building APIs. It uses type hints for automatic validation and docs, is async-first (built on Starlette + Pydantic), and is a common choice for serving GenAI/LLM endpoints.
Why FastAPI¶
- Fast — on par with Node/Go for I/O-bound work, thanks to ASGI/async.
- Type-driven — Pydantic models validate requests/responses automatically.
- Auto docs — interactive Swagger UI at
/docsand ReDoc at/redoc, free. - Editor support — type hints give autocomplete and catch errors early.
A minimal API¶
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI(title="Demo")
class AskRequest(BaseModel):
question: str
area: str = "all"
class AskResponse(BaseModel):
answer: str
citations: list[str] = []
@app.get("/health")
def health():
return {"status": "ok"}
@app.post("/ask", response_model=AskResponse)
def ask(req: AskRequest):
# ... call your logic ...
return AskResponse(answer=f"You asked about {req.area}: {req.question}")
Run it: fastapi dev main.py (or uvicorn main:app --reload) → open
http://127.0.0.1:8000/docs.
Request lifecycle¶
flowchart LR
C[Client] --> U[Uvicorn ASGI server]
U --> MW[Middleware]
MW --> DEP[Dependencies - injected]
DEP --> V[Pydantic validation]
V --> H[Path operation function]
H --> R[Response model serialize]
R --> C
Core concepts¶
| Concept | What it does |
|---|---|
| Path operations | @app.get/post/... decorators map routes to functions |
| Pydantic models | Typed request/response schemas + validation |
| Path & query params | Declared as function args with type hints |
Dependencies (Depends) |
Reusable injected logic: auth, DB sessions, config |
async def |
Non-blocking handlers for I/O (DB, HTTP, LLM calls) |
| Background tasks | Fire-and-forget work after responding |
| Routers | Split endpoints across modules (APIRouter) |
async vs sync¶
Use async def when the handler does I/O (calling an LLM, DB, or HTTP
service) with an async client — it frees the event loop to serve other requests.
Use plain def for CPU-bound or blocking-library code (FastAPI runs it in a
threadpool so it won't block the loop).
Serving an LLM / RAG endpoint¶
FastAPI is a natural front door for a GenAI service — the same role Lambda + Chainlit play in the course's SQL Assistant project:
@app.post("/chat")
async def chat(req: AskRequest):
# 1. retrieve context from a vector store (RAG)
# 2. build the prompt
# 3. await the LLM call (async client)
# 4. return the grounded answer + citations
...
Good practices: stream responses (StreamingResponse) for token-by-token UX,
set timeouts on model calls, validate/limit input size, and add rate limiting.
Validation, errors, docs¶
- Validation — invalid bodies auto-return
422with details; no manual checks. - Errors — raise
HTTPException(status_code=..., detail=...). - Docs —
/docs(Swagger) and/redocare generated from your models. - Settings — use
pydantic-settings/ env vars for config and secrets.
Deployment¶
- Uvicorn (ASGI server):
fastapi run --workers 4 main.pyoruvicorn main:app --workers 4on a VM. On Kubernetes, run one process per container and scale with replicas. (Gunicorn with Uvicorn workers is an older pattern you will still see.) - Containerize; put Nginx/ALB in front for TLS and load balancing.
- On AWS: run on ECS/Fargate, or wrap with Mangum to run on Lambda + API Gateway.
Interview questions¶
Why FastAPI over Flask?
Async-first (ASGI) for high I/O concurrency, automatic request/response validation via Pydantic type hints, and auto-generated interactive docs. Flask is sync-first (WSGI) and needs extensions for much of that.
When do you use async def vs def in a path operation?
async def for non-blocking I/O with async clients (DB, HTTP, LLM). Plain
def for blocking/CPU-bound code — FastAPI runs it in a threadpool so it
doesn't block the event loop.
What are dependencies (Depends) used for?
Reusable injected logic — auth, DB sessions, shared config, pagination — declared once and injected into any endpoint, improving testability and reuse.
How does FastAPI validate input?
You declare Pydantic models / typed params; FastAPI validates automatically and returns a structured 422 on failure — no manual parsing.
How would you serve an LLM/RAG app with FastAPI?
A /chat endpoint that retrieves context from a vector store, builds a
grounded prompt, awaits an async LLM call, and returns the answer with
citations — optionally streaming tokens via StreamingResponse.
Interview deep dive¶
60-second talking points¶
- "Type hints do the work." Pydantic validates requests/responses and generates OpenAPI docs automatically — less boilerplate, fewer bugs.
- "Async-first for I/O concurrency." Built on ASGI, so one process serves many concurrent I/O-bound requests (DB, HTTP, LLM calls).
- "Dependencies = clean, testable wiring." Auth, DB sessions, config injected
via
Depends.
Scenario & system-design questions¶
Design a streaming chat endpoint for an LLM app.
POST /chat → retrieve context (RAG) → build prompt → await the model with
an async client → return a StreamingResponse yielding tokens.
Add input-size limits, per-user rate limiting, timeouts, and a fallback if the
model call fails.
Your API blocks under load even though it's async. Why?
A blocking call inside an async def (e.g. a sync DB driver or requests)
stalls the event loop. Fix: use async clients, or move blocking work to a
threadpool (def handler / run_in_executor). CPU-bound work belongs in a
worker, not the loop.
How do you deploy FastAPI to production?
Uvicorn (fastapi run --workers N on VMs; one process per container on
K8s/ECS, scaled by replicas), behind Nginx/ALB for TLS + load balancing, or
Mangum to run on Lambda + API Gateway. Health checks, structured logging, and config via env/secrets.
How do you keep request/response contracts safe as the API evolves?
Pydantic response_model enforces the output shape; version the API
(/v1); validate inputs strictly; and rely on the auto OpenAPI schema as the
contract for clients.
Pitfalls interviewers probe¶
- Blocking I/O inside
async def(kills concurrency). - No
response_model→ leaking internal fields. - Doing heavy CPU work in the event loop.
- Skipping input validation/limits on public endpoints.
- Global mutable state instead of dependency injection.
Rapid-fire¶
| Q | A |
|---|---|
| FastAPI vs Flask? | ASGI/async + Pydantic validation + auto docs vs sync WSGI |
async def vs def? |
Async for non-blocking I/O; def (threadpool) for blocking/CPU |
What is Depends for? |
Injecting reusable logic: auth, DB, config |
| Auto docs URLs? | /docs (Swagger), /redoc |
| Stream tokens with? | EventSourceResponse (native SSE) or StreamingResponse |
| Run in prod? | fastapi run --workers N, or one Uvicorn process per container |
Deep dive: the ASGI concurrency model¶
Why FastAPI scales for I/O — and exactly how people break it.
flowchart TB
REQ[Many concurrent requests] --> LOOP[Single event loop per worker]
LOOP -->|await I/O| YIELD[Coroutine yields; loop serves others]
LOOP -->|def handler| POOL[Threadpool runs blocking code]
LOOP -->|CPU-bound| WARN[Blocks the loop - move to a worker]
- One event loop per worker process.
async defhandlers thatawaitI/O yield control so the loop serves other requests — that's the concurrency win. - The cardinal sin: a blocking call inside
async def(sync DB driver,requests,time.sleep, heavy CPU) freezes the loop and stalls every concurrent request. Use async clients, or make the handlerdef(FastAPI runsdefhandlers in a threadpool). - CPU-bound work belongs in a background worker (Celery/RQ/Arq) or a separate service — not the event loop.
- Scale out with multiple Uvicorn workers (processes); scale the loop within each for I/O. GIL means CPU parallelism needs processes, not threads.
Dependency injection in depth¶
Depends is FastAPI's backbone for clean, testable wiring.
from fastapi import Depends, HTTPException, Header
async def get_db():
async with SessionLocal() as session: # yield-based: setup + teardown
yield session
async def require_api_key(x_api_key: str = Header(...)):
if not valid(x_api_key):
raise HTTPException(401, "bad key")
return x_api_key
@app.post("/chat")
async def chat(req: AskRequest,
db=Depends(get_db),
_=Depends(require_api_key)):
...
- Yield dependencies run setup before and teardown after the request (DB sessions, transactions) — like context managers.
- Sub-dependencies compose; FastAPI resolves the graph and caches each dependency per request by default.
- Testability —
app.dependency_overridesswaps real deps for fakes in tests. This is why DI beats global state.
Serving GenAI/LLM endpoints (production shape)¶
FastAPI is the front door for the RAG/agent systems described in GenAI Topics and the Snowflake Cortex REST integration.
from collections.abc import AsyncIterable
from fastapi.sse import EventSourceResponse, ServerSentEvent
@app.post("/chat", response_class=EventSourceResponse)
async def chat(req: AskRequest, _=Depends(require_api_key)) -> AsyncIterable[ServerSentEvent]:
ctx = await retrieve(req.question) # async vector search
async for token in llm_stream(req.question, ctx): # async LLM client
yield ServerSentEvent(raw_data=token, event="token")
yield ServerSentEvent(raw_data="[DONE]", event="done")
On older FastAPI versions, use StreamingResponse and format each chunk as a
proper SSE frame (data: ...\n\n). Yielding raw tokens with a
text/event-stream media type isn't valid SSE.
Production checklist for an LLM endpoint:
- Stream tokens (
StreamingResponse/ SSE) for responsive UX. - Timeouts on every model/tool call; a hung upstream must not hang your API.
- Backpressure & concurrency caps — a semaphore limiting in-flight LLM calls so a spike doesn't exhaust tokens/budget (ties to cost optimization).
- Input limits & validation — cap prompt size; reject oversized bodies.
- Idempotency for retriable POSTs (idempotency key) so client retries don't double-charge an LLM call.
- Treat model output as untrusted — never
eval/exec it, escape before downstream use (AI Security, insecure output handling).
Security¶
- AuthN/AuthZ — OAuth2/JWT via
Depends; scope-check per route; never trust client-supplied identity fields. - Secrets —
pydantic-settings+ env/secret manager; never hardcode. - Input hardening — Pydantic validation + explicit size/rate limits; strict
response_modelso internal fields don't leak. - CORS/TLS — lock CORS to known origins; terminate TLS at the proxy/ALB.
- Rate limiting — per-key/IP (e.g. slowapi or gateway-level) to contain abuse and cost.
Observability & performance¶
- Structured logging with a request/correlation ID (middleware) so you can trace one request end-to-end.
- Metrics — latency (p50/p95/p99), error rate, in-flight requests, and for LLM apps: tokens and cost per request. Prometheus/OpenTelemetry integrate cleanly with ASGI.
- Tracing — OpenTelemetry spans across retrieve → prompt → model call to see where latency lives.
- Health/readiness —
/health(liveness) and a readiness check (deps reachable) for orchestrators. - Perf levers — async clients everywhere on the hot path, connection pooling,
cache hot reads,
orjsonresponses, and right-sizing worker count.
More system-design & production incidents¶
Each: diagnose → mitigate → prevent.
Latency spikes to seconds under moderate load though the code is async. Why?
A blocking call in an async def (sync DB/HTTP client, CPU work) is
stalling the event loop. Diagnose: profile the handler; look for non-await
I/O. Mitigate: switch to an async client or make the handler def
(threadpool); move CPU work to a worker. Prevent: lint for blocking calls,
load-test, and cap concurrency.
The LLM provider slowed down and your whole API became unresponsive. Design the fix.
No timeout/isolation on the upstream. Mitigate/prevent: per-call timeouts, a circuit breaker and retries with backoff, a concurrency semaphore so slow calls don't consume every worker, and a graceful fallback/queue. This is the reliability playbook applied at the API edge.
Design a multi-tenant LLM API that stays within a cost budget.
Per-tenant API keys (DI-checked), per-tenant rate limits and token/cost budgets, request size caps, response caching for repeat prompts, and model routing (cheap model first). Emit per-tenant cost metrics and alert on drift.
How do you deploy and roll this out safely?
Containerize; run Uvicorn (one process per container, scaled by replicas) behind ALB/Nginx (TLS, LB); run on ECS/Fargate or K8s (or Mangum on Lambda + API Gateway). Health + readiness probes, canary/blue-green rollout, structured logs, and config via env/secrets. See Kubernetes and DevOps for AI.
How do you keep the API contract stable as it evolves?
Strict response_model (no leaked fields), URL versioning (/v1), the
auto-generated OpenAPI schema as the client contract, and additive changes
over breaking ones. Deprecate old versions on a schedule.
60-second ramp checklist¶
- Explain one event loop per worker and the blocking-call trap.
- Decide
async defvsdefcorrectly for a given handler. - Use
Depends(incl. yield deps + overrides) for DB/auth/config. - Design a streaming LLM endpoint with timeouts, limits, and idempotency.
- List the security controls (auth, secrets, rate limit, response_model).
- Describe logging/metrics/tracing + health vs readiness.
- Handle the "blocking loop" and "slow upstream" incidents out loud.
Related in this site¶
- GenAI Topics · Snowflake Cortex — the systems this API fronts.
- Reliability · Kubernetes · DevOps for AI.
- AI Security — output handling and input hardening for AI endpoints.