Skip to content

Reference Architectures

Last reviewed: October 2026

Every enterprise RAG or agent platform has the same logical shape. The cloud only changes the product names and where the hard edges are. Learn the logical architecture first, then map it to each platform. In an interview you will usually be asked for the logical design before anyone cares which vector store you picked.

The logical architecture

flowchart LR
    U([Users / apps]) --> EDGE[Edge: SSO, WAF]
    EDGE --> AIGW[AI gateway<br/>authN, quotas, model routing, input guardrails]
    AIGW --> RT[Agent runtime<br/>session isolation, memory]
    RT --> KB[Knowledge layer<br/>ingest, chunk, embed, ACL metadata]
    KB --> VS[(Vector / hybrid index)]
    RT --> FM[Foundation models<br/>private endpoints]
    RT --> TGW[Tool gateway<br/>MCP/APIs, policy, OBO credentials]
    TGW --> SYS[Business systems]
    FM --> OG[Output guardrails]
    OG --> U
    subgraph Platform services
      ID[Identity: users + agent identities]
      KMS[Keys + secrets]
      OBS[OTel traces + metrics]
      AUD[(Immutable audit)]
      EVAL[Eval + red-team pipeline]
    end

The components that separate an enterprise design from a demo are the AI gateway, the tool gateway, agent identities and the audit and eval pipelines. Detail on each is in Security Architecture, RBAC, Audit Logging and Monitoring.

Platform mappings

Names change quickly

All four vendors renamed or restructured their AI platforms during 2025 and 2026. Microsoft renamed Azure AI Foundry to Microsoft Foundry. Google introduced the Gemini Enterprise Agent Platform at Cloud Next in April 2026 as the evolution of Vertex AI, and many APIs and SDKs still use vertexai names. AWS added Bedrock AgentCore, and Snowflake added Cortex Agents and Snowflake Intelligence. Check current documentation for preview and GA status in your region.

flowchart LR
    U([User]) --> CF[CloudFront + WAF]
    CF --> APIGW[API Gateway / ALB<br/>Cognito or IdP]
    APIGW --> RT[AgentCore Runtime]
    RT --> KB[Bedrock Knowledge Bases]
    KB --> VEC[(OpenSearch Serverless / Aurora pgvector / S3 Vectors)]
    RT --> BR[Bedrock models<br/>via VPC endpoint]
    BR --> GR[Bedrock Guardrails]
    RT --> GW[AgentCore Gateway + Policy]
    GW --> TOOLS[Lambda / APIs / MCP servers]
    ID[AgentCore Identity] -.-> RT
    ID -.-> GW
    RT -.-> OBS[AgentCore Observability / CloudWatch]
    BR -.-> LOG[Invocation logging -> S3 Object Lock]
Layer Service
Models Amazon Bedrock (first- and third-party models), cross-region inference profiles where residency allows
Retrieval Bedrock Knowledge Bases, metadata filtering for ACLs; OpenSearch, Aurora pgvector or S3 Vectors
Agents Bedrock AgentCore, a set of modular services that includes Runtime, Memory, Gateway, Identity, Policy (GA March 2026, Cedar-based), Code Interpreter, Browser, Observability, Evaluations and Registry
Guardrails Bedrock Guardrails: content filters, prompt attacks, denied topics, word filters, sensitive information filters, contextual grounding, Automated Reasoning checks. Also usable standalone via the ApplyGuardrail API
Security IAM, KMS, Secrets Manager, PrivateLink VPC endpoints, SCPs to restrict models and regions
Evidence CloudTrail (control plane), Bedrock invocation logging, S3 Object Lock

Watch for: cross-region inference profiles can route requests to other regions in the same geography. Confirm this matches your residency commitments.

flowchart LR
    U([User]) --> FD[Front Door + WAF]
    FD --> APIM[API Management<br/>AI gateway: token limits, routing]
    APIM --> AG[Foundry Agent Service]
    AG --> AIS[(Azure AI Search<br/>hybrid + vector, security filters)]
    AG --> FM[Foundry Models<br/>incl. Azure OpenAI, private endpoint]
    FM --> CS[Guardrails / Content Safety<br/>Prompt Shields]
    AG --> TOOLS[MCP servers / Functions / Logic Apps]
    ENT[Entra ID + agent identities] -.-> AG
    AG -.-> AI[Application Insights / Azure Monitor]
    PV[Purview] -.-> AIS
Layer Service
Models Microsoft Foundry Models, including Azure OpenAI; regional or data-zone deployments for residency
Retrieval Azure AI Search (hybrid, semantic ranking, security-trimming filters)
Agents Foundry Agent Service: per-agent Entra identities, MCP tools, private networking, tracing to Application Insights
Guardrails Foundry guardrails and content filters built on Azure AI Content Safety, including Prompt Shields for direct and indirect (cross-prompt) injection
Gateway Azure API Management AI gateway policies: token rate limits, load balancing across deployments, semantic caching
Security Entra ID RBAC and on-behalf-of flow, Key Vault, Private Link, Azure Policy, Microsoft Purview for data governance

Watch for: global deployment types can process requests outside your chosen region. Pick regional or data-zone deployments when residency matters.

flowchart LR
    U([User]) --> LB[Cloud Load Balancing + Cloud Armor]
    LB --> RUN[Cloud Run / GKE<br/>IAP or IdP]
    RUN --> AR[Agent Runtime<br/>ADK agents]
    AR --> VS[(Vector Search / RAG Engine / AlloyDB)]
    AR --> GEM[Gemini + Model Garden models]
    AR --> AGW[Agent Gateway<br/>+ Model Armor]
    AGW --> TOOLS[MCP / APIs / A2A agents]
    AID[Agent Identity] -.-> AR
    AR -.-> OBS[Cloud Trace / Logging / Monitoring]
    VPCSC[VPC Service Controls] -.-> GEM
Layer Service
Platform Gemini Enterprise Agent Platform (evolution of Vertex AI): Build, Scale, Govern, Optimize
Models Gemini models and Model Garden partner and open models
Retrieval Vector Search, RAG Engine, or AlloyDB / BigQuery vector search for governed data
Agents Agent Development Kit (ADK), Agent Runtime, Memory Bank
Governance Agent Identity (per-agent cryptographic identity), Agent Gateway (policy between agents and tools; check release status), Model Armor (prompt injection, tool poisoning, sensitive data leakage)
Security Cloud IAM, VPC Service Controls perimeters, CMEK, Private Service Connect, organization policies restricting models and locations
Evidence Cloud Audit Logs (enable Data Access logs explicitly), Cloud Logging buckets with retention locks

Watch for: Data Access audit logs are off by default for most services. Turn them on for AI services in scope.

flowchart LR
    U([User]) --> SI[Snowflake Intelligence / Streamlit / API]
    SI --> CAG[Cortex Agents]
    CAG --> CSS[Cortex Search<br/>unstructured RAG]
    CAG --> CAN[Cortex Analyst<br/>semantic views -> SQL]
    CSS --> DATA[(Governed tables + stages)]
    CAN --> DATA
    CAG --> AISQL[Cortex AISQL functions / LLMs]
    GRD[Cortex AI Guardrails] -.-> CAG
    HZN[Horizon: RBAC, row access, masking, tags] -.-> DATA
    CAG -.-> OBS[AI observability + account usage]
Layer Service
Models Cortex LLM functions and AISQL (e.g. AI_COMPLETE) on models hosted inside Snowflake's boundary
Retrieval Cortex Search (hybrid search over text), Cortex Analyst for text-to-SQL over semantic views
Agents Cortex Agents orchestrate Search, Analyst and custom tools; Snowflake Intelligence is the end-user surface
Guardrails Cortex AI Guardrails (GA April 2026, run-time prompt-injection and jailbreak protection)
Governance Existing RBAC, row access and masking policies, tags, access history: the LLM path inherits them when it runs with the caller's rights
Evidence Account usage views for Cortex consumption, access history, query history

Watch for: the account parameter CORTEX_ENABLED_CROSS_REGION controls whether Cortex can route inference to other regions. Snowflake's 2026_06 behavior-change bundle sets a geography-scoped default (for example AWS_EU) for accounts that have never set the parameter, with some regulated account types excluded. Set it explicitly at the account level, either DISABLED or the geography you have approved, rather than relying on the default.

Choosing a platform

Situation Lean Why
Workloads and data already on AWS; need model choice AWS Bedrock + AgentCore Broad model catalog, IAM-native, managed agent runtime and tool policy
Microsoft 365 / Entra estate; OpenAI models; Copilot integration Microsoft Foundry Entra identity end to end, APIM AI gateway, Purview
BigQuery-centric analytics; Gemini; Google Workspace Gemini Enterprise Agent Platform Gemini models, agent identity and gateway, VPC Service Controls
Governed data lives in Snowflake; no-egress requirement Snowflake Cortex Inference next to governed data; existing policies apply
Multi-cloud or strict portability Cloud-neutral runtime (e.g. LangGraph on Kubernetes) + gateway + OTel You give up managed security features for portability; budget for building them

Hybrid is common: governed analytics on Snowflake or Databricks, plus agents on a hyperscaler calling them through a tool gateway. The design question is where identity and policy are enforced at each hop.

Failure modes and anti-patterns

Architecture smells

  • Choosing by model leaderboard alone. Identity, network isolation, residency and evidence usually matter more for an enterprise than a few benchmark points.
  • Managed agent runtime without a tool policy layer. The runtime is isolated, but the agent can still call anything its role allows.
  • A vector store outside the governance perimeter. Data is copied from a governed warehouse into an unmanaged index with none of its policies.
  • Assuming "in-account" means "in-region" without checking cross-region routing settings.
  • Building on preview services for regulated production without an exit plan.

How interviewers probe this

Design an enterprise knowledge assistant on AWS for 30,000 employees across the EU and US.

Cover SSO, an AI gateway with per-user quotas, regional deployments for EU and US data, Knowledge Bases with ACL metadata filters, Guardrails, AgentCore Identity and Gateway for tools, invocation logging to Object Lock buckets, and OTel tracing. Then the trade-offs: per-region indexes against a global index, and how ACL sync works.

Why would you put an LLM workload inside Snowflake instead of calling a hyperscaler model?

Governance comes for free: row access and masking policies apply, no data copy or egress, one bill. Costs: smaller model and tool catalog, credit-based pricing and less control over serving. Mention the cross-region inference setting.

The bank's security team will only approve one entry point for all model traffic. Draw it.

An AI gateway (APIM, a self-hosted gateway, or an equivalent) with authentication, model allow-list, quotas, input guardrails, logging and routing to regional endpoints. Direct provider access is blocked with org policies and network rules. Explain how agents' tool calls go through a separate tool gateway.

How would you make this portable across clouds?

Use an open agent framework, OTel conventions, MCP for tools, an abstraction at the gateway, and data contracts for retrieval. Be honest about what you give up: managed identity, guardrails and evaluations, which you then have to build or buy.

Checklist

  • Logical architecture agreed before choosing services
  • Single AI gateway for model traffic; direct access blocked
  • Tool gateway with policy and on-behalf-of credentials
  • Residency verified for inference, embeddings, caches, traces and logs
  • Guardrails on input and output; prompt-attack detection for indirect injection
  • Audit evidence in immutable storage; OTel traces
  • Preview-service dependencies listed, with exit plans

Further reading