Model Gateway Routing

---
name: model-gateway-routing
description: |
  Model gateway / LLM router architecture: a control point in front of multiple
  models/providers for routing (cost/quality/latency), fallback, rate limiting,
  caching, observability, and governance. Architect-level, multi-provider.

  USE WHEN: designing an LLM gateway/router, "model router", "LLM gateway",
  "multi-provider", "fallback", "cost routing", "LiteLLM", "Envoy AI Gateway",
  semantic cache, central key/quota/observability for LLM calls.

  DO NOT USE FOR: single-engine serving (use `inference-serving-topology`);
  edge/cascade (use `hybrid-edge-cloud`); agent orchestration (use `agentic-architecture`).
allowed-tools: Read, Grep, Glob
---
# Model Gateway / Routing

A gateway is a single control point between apps and many models/providers.
It turns "which model?" and cross-cutting concerns into infrastructure.

## What it centralizes (the reasons to build/buy one)
- **Routing**: pick a model per request by **cost / quality / latency / context
  length / capability**, or A/B and canary new models.
- **Fallback & resilience**: retry/failover across providers on error or rate
  limit; circuit-break a failing provider.
- **Cost control**: per-team/app budgets, quotas, and **cost attribution**;
  route cheap queries to cheap models.
- **Caching**: exact + **semantic cache** to skip duplicate/near-duplicate calls.
- **Security/governance**: central API-key custody, PII redaction, audit logs,
  policy (which teams may call which models).
- **Observability**: latency/tokens/cost/error metrics in one place.

Implementations: **LiteLLM**, **Envoy AI Gateway**, cloud AI gateways, or custom.

## Design decisions
- **Routing policy**: static (rules) vs learned/heuristic (route by predicted
  difficulty). Keep it explainable; mind added hop latency.
- **Sync vs streaming**: must pass through token streaming with low overhead.
- **Statelessness**: keep the gateway stateless + horizontally scalable; push
  state (cache, budgets) to fast stores.
- **Failure semantics**: define what happens when all providers fail.

## When to recommend
- Multiple models/providers, multiple teams, real cost/governance needs → yes.
- Single model, single team, prototype → a gateway is premature; call the model
  directly and add the gateway when the second model/provider/team appears.