DevOps Interview Q&A — Advanced & Scenario-Based¶
Last reviewed: October 2026
Senior DevOps / platform questions: CI/CD design, IaC, containers and Kubernetes, observability, incident response, and cloud deployment patterns. Study at a glance, then open each question for depth.
How to use this page
Skim the 60-second talking points and rapid-fire for recall, then drill into the collapsible questions. Finish with the self-quiz. Related: AWS Interview Q&A.
Study checklist¶
Can you explain each without notes?
- "Build once, promote the artifact" and why
- Externalized config to avoid drift
- Blue/green vs canary vs rolling
- IaC plan/apply, state locking, drift
- Requests vs limits in Kubernetes
- Debugging CrashLoopBackOff
- Secrets management done right
- The three pillars of observability
- SLI / SLO / error budget
- Incident response order (restore first)
- Software supply-chain security (OIDC, SBOM, signing, pinned actions)
- Shipping LLM features: prompts and models as versioned config
60-second talking points¶
- "Everything as code, everything reproducible." Infra, pipelines, and config in version control so environments are rebuildable, not hand-crafted.
- "Immutable deployments beat in-place mutation." Build an artifact once, promote the same artifact through environments.
- "Automate the path to prod, gate it with checks." Fast CI feedback, controlled promotion with approvals and rollbacks.
Core concepts — simple, then the nuance¶
Build once, promote: explain it simply, then go deep
Simple: Build the artifact (container image) one time, then move that exact same image through dev → test → prod. Don't rebuild per environment.
The nuance: Rebuilding per environment risks a different artifact reaching prod than you tested (different base image, dependency versions, build flags). Promotion means the tested bits are the shipped bits. Environment differences live in externalized config (env vars, parameter store, secrets manager), never baked into the image.
SLI/SLO/error budget: simple, then deep
Simple: SLI = a number you measure (e.g. % successful requests). SLO = the target for it (99.9%). Error budget = how much you're allowed to miss (0.1%).
The nuance: The error budget turns reliability into a shared decision: budget healthy → ship features fast; budget burning → stop and fix reliability. It aligns dev and ops on data instead of opinion, and it's why you alert on SLO burn (symptoms users feel) rather than every low-level metric.
CI/CD¶
Worked scenario: a deploy broke prod
Symptom: New release deployed, error rate spikes, users affected.
Reasoning / order of operations: 1. Restore service first — roll back to the last-good artifact / flip blue-green. Confirm recovery on dashboards. Don't debug a burning prod. 2. Then investigate with logs/traces/metrics around the change window. 3. Prevent recurrence: smaller/more frequent deploys, canary with automated rollback on error-rate/latency thresholds, better pre-prod tests, feature flags to decouple deploy from release. 4. Blameless postmortem to fix the systemic gap, not the person.
Outcome: "Rollback first, investigate second. Then make the next deploy safer with canary + flags."
Design a CI/CD pipeline for a containerized service.
On PR: lint, unit tests, security scan, build. On merge: build an immutable image tagged by commit SHA, push, deploy dev, run integration/smoke tests, then promote the same image to test and prod behind approvals. Prod uses a safe strategy (blue/green or canary) with health checks and one-command rollback. Never rebuild per environment — promote the tested artifact.
How do you promote across environments without config drift?
Same artifact, externalized config per environment (env vars, parameter store, secrets manager) — never baked into the image. Infra as code applied per environment from the same templates. Branch/tag strategy controls eligibility so promotion is deterministic.
A deploy broke prod. Walk me through response and prevention.
Response: roll back first (redeploy last-good / flip blue-green), confirm recovery, then investigate. Prevention: smaller/frequent deploys, canary with automated rollback, better pre-prod coverage, feature flags to decouple deploy from release, and a blameless postmortem.
Feature flags vs deploying — why decouple them?
Deploying ships code; a feature flag controls whether it's active. Decoupling lets you deploy dark, enable for a % of users, roll back a feature instantly without a redeploy, and test in prod safely. It shrinks blast radius and separates "is it deployed" from "is it on."
Infrastructure as Code¶
How do you keep IaC safe and reviewable?
Plan before apply (review the diff in PR), remote state with locking to
prevent concurrent corruption, modularize for reuse, pin provider/module
versions, separate state per environment, gate prod apply behind approval, and
detect drift on a schedule. Never edit cloud resources by hand outside IaC.
Someone changed a resource in the console and IaC wants to revert it. What happened?
Drift: live state no longer matches code, so the next plan proposes to reconcile (often destructively). Fix: import the manual change into code, or revert the console change, then re-plan. Prevent it by locking down console write access in prod and requiring changes through IaC + PR.
Blue/green vs canary vs rolling — trade-offs?
Rolling: replace instances gradually; simple, but mixed versions during rollout and slower rollback. Blue/green: full standby env, instant switch and rollback, but double resources briefly. Canary: route a small % to the new version, watch metrics, ramp or abort; safest but needs good routing + metrics. Choose by risk tolerance and infra cost.
How do you manage state for a team so two applies don't collide?
Remote backend with state locking, so a second apply waits or fails fast
instead of corrupting state. On AWS, Terraform 1.10+ supports native S3
locking (use_lockfile = true), and the older DynamoDB lock table is now
deprecated; other backends (Terraform Cloud/HCP, GCS, azurerm) lock natively. Separate state per
environment/component to shrink blast radius, restrict who can apply to prod, and
never store state locally for shared infra.
How do you secure the CI/CD supply chain?
No long-lived cloud keys in CI: use OIDC federation (e.g. GitHub Actions →
AWS IAM role) with short-lived, branch-scoped credentials. Pin third-party
actions and base images by digest/SHA, not mutable tags (the 2025
tj-actions/changed-files compromise leaked secrets from thousands of repos
through a retagged action). Generate an SBOM, scan dependencies and images,
sign artifacts (Sigstore/cosign) and verify signatures and provenance
(SLSA attestations) at deploy time with admission policies. Least-privilege
tokens per job, protected branches and required reviews for workflow changes,
and isolated runners for untrusted PRs.
How do you deploy an LLM-backed feature safely?
Treat prompts, model IDs, tool definitions and retrieval config as versioned artifacts in the same build-once-promote flow. Gate promotion on an offline eval suite (quality, safety, cost per request) as well as unit tests. Roll out with a flag or canary and compare online metrics (task success, user feedback, refusal and error rates, p95 latency, tokens per request) against the old version. Pin model versions so a provider update doesn't ship silently, keep a fallback model, and trace every call so rollbacks are evidence-based.
Containers & Kubernetes¶
A pod keeps restarting (CrashLoopBackOff). How do you debug?
kubectl describe pod (events: OOMKilled? failed probe? image pull?),
kubectl logs --previous for the crashed container. Common causes: failing
liveness probe (too aggressive timing), OOMKilled (memory limit too low),
missing config/secret, or a real startup crash. Fix the root cause — adjust probe
timing, raise limits, supply config — don't just bump restarts.
Requests vs limits — why do both matter?
Requests drive scheduling (guaranteed resources, bin-packing). Limits cap usage (CPU throttled; memory over-limit → OOMKilled). Set requests to typical usage, limits to a safe ceiling. No requests → poor scheduling; no limits → noisy neighbor starves the node. Watch for CPU throttling from tight limits.
How do you manage secrets in Kubernetes properly?
Don't commit secrets or bake them into images. Use a secrets manager (cloud KMS-backed) via an operator/CSI driver to mount them, or sealed/encrypted secrets in Git. Enable etcd encryption at rest, scope RBAC so pods read only what they need, and rotate regularly.
Liveness vs readiness probe — what's the difference?
Liveness = "is the app alive?" — fail → restart the container. Readiness = "can it serve traffic now?" — fail → remove from the load balancer but don't restart. Aggressive liveness timing causes false restarts; readiness handles slow startup and transient unavailability gracefully.
Observability & reliability¶
What do you monitor, and how do you avoid alert fatigue?
Three pillars: metrics (RED/USE — rate, errors, latency, saturation), logs (structured, correlated), traces (request flow across services). Alert on symptoms users feel (SLO burn, error rate, latency), not every low-level metric. Page only on actionable, urgent conditions; everything else is a dashboard/ticket. Tie alerts to SLOs and error budgets.
Explain SLI/SLO/error budget and how it changes behavior.
SLI = measured indicator (e.g. % successful requests). SLO = the target (99.9%). Error budget = allowed failure (0.1%). Budget healthy → ship fast; budget burning → slow down and prioritize reliability. It turns "how much reliability" into a shared, data-driven decision.
You get paged for high latency. Walk me through triage.
Confirm scope on dashboards (which service/region/endpoint, since when). Correlate with recent changes (deploys, config, traffic). Use traces to find the slow hop (DB? downstream? GC?). Mitigate first (rollback, scale out, shed load), then root-cause. Communicate status. Postmortem after.
Rapid-fire¶
| Q | A |
|---|---|
| Immutable infra? | replace instances/images rather than mutating in place |
| Blue/green? | two envs, instant switch + rollback |
| Canary? | small % traffic to new version, ramp on good metrics |
| IaC drift? | live state diverges from code |
| CrashLoopBackOff top causes? | failed probe, OOMKilled, bad config, startup crash |
| Requests vs limits? | scheduling vs capping (throttle/OOM) |
| Liveness vs readiness? | restart-if-dead vs remove-from-LB-if-not-ready |
| RED metrics? | Rate, Errors, Duration |
| Error budget? | allowed unreliability = 1 − SLO |
| Feature flag vs deploy? | activation control vs shipping code |
| CI → cloud auth? | OIDC federation, short-lived role credentials |
| Pin GitHub Actions how? | full commit SHA, not a tag |
Pitfalls interviewers probe¶
- Rebuilding artifacts per environment instead of promoting one.
- Baking environment config/secrets into images.
- Editing prod resources in the console (drift).
- Liveness probes too aggressive → false restarts.
- Alerting on causes not symptoms → fatigue.
- No rollback plan; investigating before restoring service.
Self-quiz¶
- Design a CI/CD pipeline for a containerized service.
- Why promote one artifact instead of rebuilding per env?
- A prod deploy broke — what's your order of operations?
- Blue/green vs canary vs rolling — pick one for a risky change.
- Debug a CrashLoopBackOff step by step.
- Requests vs limits — what breaks if you omit each?
- Explain error budgets and how they change shipping behavior.
- You're paged for latency — how do you triage?
Cross-links
Related: AWS Interview Q&A · AI Engineer Interview Q&A