Security & Governance¶
Last reviewed: October 2026
GenAI adds new risks on top of normal application security: prompt injection, data leakage through prompts and retrieval, agents taking unintended actions, unbounded consumption, and unclear data flows to third-party models. This page covers the threat model, the controls, and how they map to the frameworks auditors and interviewers ask about. For identity-specific depth, see AI Security.
The organising principle: the model is not a security boundary. Anything the model can be talked into, an attacker can talk it into. Enforce controls in code, identity and infrastructure, and use the model only as one layer.
Defence in depth¶
flowchart TB
IN[User input] --> G1[Input guardrails: injection, PII, toxicity, size limits]
G1 --> APP[App logic]
APP --> RET[Retrieval - tenant + ACL filtered]
APP --> TOOLS[Tools - scoped creds, approvals]
APP --> LLM[LLM - in governed boundary]
EXT[External content: web, email, docs] -. untrusted .-> RET
LLM --> G2[Output guardrails: PII redaction, safety, format, link checks]
G2 --> OUT[Response]
AUDIT[(Audit log / traces)] -.records.-> APP
IAM[Identity & RBAC] -.authorizes.-> APP
IAM -.authorizes.-> TOOLS
IAM -.authorizes.-> RET
Threat model: OWASP Top 10 for LLM applications¶
The OWASP list (2025 edition) is the most common shared vocabulary.
| OWASP risk | What it looks like | Primary controls |
|---|---|---|
| LLM01 Prompt injection | Direct ("ignore previous instructions") or indirect via retrieved docs, emails, web pages | Treat content as data, least-privilege tools, approvals, output checks |
| LLM02 Sensitive information disclosure | Model reveals PII, secrets or another tenant's data | Don't put secrets in prompts; ACL-filtered retrieval; output redaction |
| LLM03 Supply chain | Compromised models, packages, MCP servers, datasets | Pin and verify artefacts, vet third-party tools and servers |
| LLM04 Data and model poisoning | Malicious content enters the index or fine-tuning data | Source allow-lists, ingestion review, provenance metadata |
| LLM05 Improper output handling | Model output executed as SQL, shell, HTML | Treat output as untrusted input; parameterise, sandbox, escape |
| LLM06 Excessive agency | Agent has more tools, permissions or autonomy than needed | Scoped tools, read/write split, human approval |
| LLM07 System prompt leakage | Prompt reveals internal rules or credentials | Assume the system prompt is public; no secrets in it |
| LLM08 Vector and embedding weaknesses | Cross-tenant retrieval, embedding inversion | Per-tenant namespaces or enforced filters, access control on the store |
| LLM09 Misinformation | Confident, wrong answers acted on by users | Grounding, citations, abstention, human review for high stakes |
| LLM10 Unbounded consumption | Cost or resource exhaustion, model extraction | Rate limits, token quotas, budgets, abuse monitoring |
Key controls¶
| Risk | Control |
|---|---|
| Prompt injection | Treat retrieved/user content as data, not instructions; least-privilege tools; output checks |
| Data leakage | No secrets/PII in prompts unless required; redact; keep data in-boundary (e.g. Bedrock in-account, Snowflake Cortex) |
| Unauthorized access | RBAC; row/column security on retrieval; tenant filters enforced in vector search |
| Hallucinated actions | Human-in-the-loop for writes; confirmation + limits; idempotent tools |
| Unsafe output | Output guardrails (PII, toxicity, denied topics); escape before rendering |
| Exfiltration via output | Block or proxy images and links to unknown domains in rendered output |
| Cost abuse | Per-user and per-tenant token quotas, max output, agent budgets |
| No accountability | Audit logging and tracing of prompts, retrieved docs, tool calls, decisions |
The "lethal trifecta" for agents¶
Simon Willison's framing: an agent becomes high risk when it combines access to private data, exposure to untrusted content, and a way to send data out (email, HTTP, writing to shared locations). If all three are present, assume injection will eventually succeed and remove at least one leg, or require human approval on the outbound path.
Agent and tool permissions¶
# Example policy for an MCP tool server (illustrative)
tools:
search_kb: { access: read, scope: "tenant:{session.tenant_id}" }
get_invoice: { access: read, scope: "customer:{session.user_id}" }
issue_refund:
access: write
max_amount_usd: 100
requires_approval: true # human in the loop above this tool
idempotency_key: required
send_email:
access: write
allowed_domains: ["@acme.com"] # removes the exfiltration leg
Principles: the session identity, not the model, decides scope; credentials are per tool and short-lived; write tools are explicit, limited and logged; third- party MCP servers are reviewed like any dependency.
Data governance¶
- Know your data flow: what leaves your boundary, to which provider and region, how long it is retained, and whether it is used for training. Prefer in-account (Bedrock, Vertex AI, Azure) or in-platform inference (Snowflake Cortex, Databricks) to reduce egress.
- Classification and masking: tag PII at the source; mask in retrieval sources; carry classification through to outputs and logs.
- Least privilege: scope model, tool and data access to the minimum.
- Logs are sensitive data too: traces contain prompts and outputs. Apply retention limits, access control and redaction to observability stores.
- Right to delete: deletion must propagate to vector indexes, caches, traces and any fine-tuning datasets.
Guardrail options¶
- Cloud-native: Amazon Bedrock Guardrails (denied topics, content filters, PII redaction, contextual grounding checks); equivalent safety filters on Vertex AI and Azure.
- Open frameworks: NVIDIA NeMo Guardrails for programmable rails; open safety classifiers such as Llama Guard or gpt-oss-safeguard.
- Your own checks: regex and format validation, allow-lists, schema validation, and an LLM safety classifier.
Place cheap deterministic checks inline; run expensive model-based checks in parallel with generation or on a sample, depending on risk.
Compliance and frameworks¶
| Framework | What it asks for | GenAI touchpoints |
|---|---|---|
| GDPR / CCPA | Lawful basis, minimisation, rights to access and delete | PII in prompts, retention, deletion from indexes and logs |
| SOC 2 / ISO 27001 | Access control, logging, change management | Prompt and model versioning, audit trails, vendor review |
| ISO/IEC 42001 | AI management system | Risk assessment, roles, lifecycle controls |
| NIST AI RMF | Govern, Map, Measure, Manage | Risk register, eval evidence, incident response |
| EU AI Act | Risk-tiered obligations, phased in over several years | Use-case classification, transparency, documentation for high-risk systems |
| Sector rules (finance, health, energy) | Domain-specific | Human review for regulated decisions, explainability |
Keep humans in the loop for regulated or consequential decisions, and keep the evidence (eval results, approvals, logs) that shows you did.
Common failure modes¶
| Failure | Why it happens | Fix |
|---|---|---|
| "The system prompt says don't reveal X" as the only control | Treating the model as a boundary | Enforce in code; assume prompts leak |
| Tenant filter applied after retrieval | Convenience | Filter inside the query; test with cross-tenant probes |
| Agent with a broad service account | Fast prototyping | Per-tool, per-user scoped credentials |
| Full prompts logged forever | Default observability settings | Redact, retain briefly, restrict access |
| Unreviewed third-party MCP server | Treated as config, not code | Vendor and code review, pinning, sandboxing |
How interviewers probe this¶
How would you defend an email-reading agent against indirect prompt injection?
Accept that detection is imperfect. Break the trifecta: restrict outbound actions (allow-listed recipients, approval for sends), keep read and write tools separate, mark email content as data, monitor for anomalous tool sequences, and red-team with injected emails in the eval suite.
How do you prevent cross-tenant data leakage in a RAG system?
Server-side tenant identity, per-tenant namespaces or mandatory filters in the search call, ACL sync on permission changes, cache keys that include tenant, and automated cross-tenant tests in CI.
Legal asks what happens to customer data sent to the model. What do you need to know?
Provider data terms (retention, training use, zero-retention options), region of processing, sub-processors, what is logged on your side, and deletion paths. A strong answer produces a data-flow diagram.
Where do guardrails go, and what do they cost?
Input and output, plus tool-call policy checks. Trade-offs: latency, false positives blocking legitimate users, and cost of model-based classifiers. Mentions tuning thresholds against labeled data.
How would you prepare a GenAI system for an audit?
Inventory of models and use cases with risk classification, versioned prompts and models, eval evidence, access controls, audit logs, incident process, and mapping to NIST AI RMF or ISO/IEC 42001 controls.
Further reading¶
- OWASP Top 10 for LLM Applications
- NIST AI Risk Management Framework
- MITRE ATLAS
- EU AI Act (Regulation (EU) 2024/1689)
- ISO/IEC 42001
- Amazon Bedrock Guardrails
- Greshake et al., Indirect Prompt Injection
- Related here: AI Security · Identity & API security · AgentCore Identity & Gateway · Snowflake Cortex governance · Snowflake governance · Bedrock · Observability & Eval · AWS security (Interview Q&A)