Istio

---
name: istio
description: "Expert coverage of Istio service mesh across all supported versions: ambient mesh (ztunnel/waypoint), sidecar mode, istiod control plane, Envoy data plane, VirtualService, DestinationRule, security policies, observability, and Kubernetes Gateway API. Use for \"Istio\", \"istiod\", \"VirtualService\", \"DestinationRule\", \"ambient mesh\", \"ztunnel\", \"waypoint\", \"Envoy sidecar\", \"Istio Gateway\", \"PeerAuthentication\"."
license: MIT
---

# Istio

This skill covers Istio across all supported versions (1.23 through 1.27), including:

- Ambient mesh architecture (ztunnel, waypoint proxies, HBONE)
- Sidecar mode (Envoy sidecar injection, iptables interception)
- istiod control plane (Pilot, Citadel, Galley merged into single binary)
- Traffic management (VirtualService, DestinationRule, Gateway, Gateway API)
- Security (PeerAuthentication, AuthorizationPolicy, RequestAuthentication)
- Observability (Prometheus metrics, Kiali, Jaeger, distributed tracing)
- Multi-cluster and multi-network configurations

When a question is version-specific, read the matching file in the Version-specific guidance section below. When the version is unknown, provide guidance based on the latest stable release.

## How to Approach Tasks

1. **Classify** the request:
   - **Traffic management** -- Load `references/best-practices.md` for routing patterns, canary, circuit breaking
   - **Architecture** -- Load `references/architecture.md` for ambient vs sidecar, istiod, ztunnel, HBONE
   - **Security** -- Apply mTLS, AuthorizationPolicy, JWT patterns
   - **Observability** -- Metrics, tracing, Kiali dashboard
   - **Installation/migration** -- Ambient profile, sidecar-to-ambient migration

2. **Determine mode** -- Is the user running ambient mesh or sidecar mode? This fundamentally changes the architecture and troubleshooting approach. Ambient is GA since 1.24.

3. **Load context** -- Read the relevant reference file below.

4. **Analyze** -- Apply Istio-specific reasoning with awareness of Envoy proxy behavior.

5. **Recommend** -- Provide actionable guidance with YAML manifests, istioctl commands, and validation steps.

6. **Verify** -- Suggest validation steps (`istioctl analyze`, `istioctl proxy-status`, Kiali dashboard).

## Architecture: Ambient Mode (GA since 1.24)

Ambient mesh is the sidecar-less architecture and the recommended approach for new deployments.

### Components

**ztunnel (L4 per-node proxy):**
- Rust-based DaemonSet, one pod per node
- Handles mTLS encryption/decryption for all pod traffic on the node
- Enforces L4 authorization policies
- Emits L4 telemetry (connection-level metrics)
- Routes traffic via HBONE (HTTP/2 CONNECT tunnels) between nodes
- No per-pod overhead -- shared across all pods on the node

**Waypoint proxy (L7 per-namespace/service):**
- Envoy-based Deployment, deployed only when L7 features are needed
- Handles HTTP routing, retries, timeouts, circuit breaking
- Enforces L7 authorization policies (method, path, headers)
- Emits L7 telemetry (request-level metrics)
- Scoped to a namespace or specific services

### Enabling Ambient Mode

```bash
# Install Istio with ambient profile
istioctl install --set profile=ambient

# Enable ambient for a namespace (L4 only)
kubectl label namespace production istio.io/dataplane-mode=ambient

# Deploy waypoint for L7 features (optional)
istioctl waypoint apply --namespace production

# Verify
kubectl get pods -n istio-system
# istiod-xxxxx    Control plane
# ztunnel-xxxxx   DaemonSet on each node (istio-system)

kubectl get pods -n production
# waypoint-xxxxx  Waypoint proxy (if deployed)
```

### HBONE (HTTP-Based Overlay Network Environment)

ztunnel creates HTTP/2 CONNECT tunnels between nodes to carry mTLS-encrypted traffic:

```
Pod A (Node 1) --> ztunnel (Node 1) --HBONE tunnel (mTLS)--> ztunnel (Node 2) --> Pod B (Node 2)
```

HBONE is transparent to applications. It encapsulates TCP traffic in HTTP/2 CONNECT, providing mTLS without per-pod proxy overhead.

## Architecture: Sidecar Mode

The traditional Istio architecture with per-pod Envoy sidecars.

### Sidecar Injection

```bash
# Auto-injection via namespace label
kubectl label namespace production istio-injection=enabled

# Manual injection
istioctl kube-inject -f deployment.yaml | kubectl apply -f -

# Verify sidecar is present
kubectl get pod mypod -o jsonpath='{.spec.containers[*].name}'
# Returns: app istio-proxy
```

### Traffic Interception

An init container (`istio-init`) configures iptables rules that redirect all inbound/outbound TCP traffic through the Envoy sidecar:

```
App container (listens on :8080)
  |
  iptables REDIRECT --> Envoy sidecar (:15001 outbound, :15006 inbound)
  |
  Envoy applies routing, mTLS, policies --> destination
```

## Traffic Management

### VirtualService

```yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.production.svc.cluster.local
  http:
  # Canary: 10% to v2
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10
    timeout: 5s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: "5xx,connect-failure"
  # Header-based routing
  - match:
    - headers:
        x-user-role:
          exact: beta-tester
    route:
    - destination:
        host: myapp
        subset: v2
```

### DestinationRule

```yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 1000
    loadBalancer:
      simple: ROUND_ROBIN
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
```

### Gateway (Istio API)

```yaml
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: main-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: myapp-tls-cert
    hosts:
    - "*.example.com"
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "*.example.com"
```

### Kubernetes Gateway API

Istio supports the standard Kubernetes Gateway API (recommended for new deployments):

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: production
spec:
  gatewayClassName: istio
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - name: myapp-tls

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: myapp-v1
      port: 8080
      weight: 90
    - name: myapp-v2
      port: 8080
      weight: 10
```

## Security

### mTLS (PeerAuthentication)

```yaml
# Strict mTLS for entire namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

# Permissive for specific service (migration period)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: legacy-service
  namespace: production
spec:
  selector:
    matchLabels:
      app: legacy
  mtls:
    mode: PERMISSIVE
```

### AuthorizationPolicy

```yaml
# Default deny all
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production
spec: {}

# Allow specific service-to-service communication
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-api
  namespace: production
spec:
  selector:
    matchLabels:
      app: api
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/production/sa/frontend"]
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]
```

### RequestAuthentication (JWT)

```yaml
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: jwt-auth
spec:
  selector:
    matchLabels:
      app: api
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"
    audiences: ["api.example.com"]
    forwardOriginalToken: true
```

## Observability

### Built-in Metrics

Istio proxies automatically emit Prometheus metrics:

| Metric | Type | Description |
|---|---|---|
| `istio_requests_total` | Counter | Total requests with labels (source, dest, method, code) |
| `istio_request_duration_milliseconds` | Histogram | Request latency distribution |
| `istio_tcp_sent_bytes_total` | Counter | TCP bytes sent |
| `istio_tcp_received_bytes_total` | Counter | TCP bytes received |
| `istio_tcp_connections_opened_total` | Counter | TCP connections opened |

### Dashboards

```bash
istioctl dashboard kiali         # Service topology + health
istioctl dashboard grafana       # Pre-built Istio dashboards
istioctl dashboard jaeger        # Distributed tracing
istioctl dashboard prometheus    # Raw metrics
```

### Distributed Tracing

Applications must propagate these headers for trace correlation:
- `traceparent` (W3C TraceContext)
- `x-request-id`
- `x-b3-traceid`, `x-b3-spanid`, `x-b3-parentspanid`, `x-b3-sampled` (B3)

## Troubleshooting

```bash
# Analyze configuration for issues
istioctl analyze -n production

# Check proxy sync status
istioctl proxy-status

# View proxy configuration
istioctl proxy-config routes <pod-name> -n production
istioctl proxy-config clusters <pod-name> -n production
istioctl proxy-config endpoints <pod-name> -n production

# Check proxy logs
kubectl logs <pod-name> -c istio-proxy -n production

# Debug connection issues
istioctl proxy-config log <pod-name> --level debug

# Verify mTLS status
istioctl authn tls-check <pod-name> <destination-service>
```

## Common Pitfalls

1. **PERMISSIVE mode left in production**: After migration, switch to STRICT mTLS to prevent plaintext traffic.
2. **VirtualService without DestinationRule subsets**: Traffic splitting requires both VirtualService (weights) and DestinationRule (subset definitions).
3. **Not propagating trace headers**: Without application-level header propagation, distributed traces are disconnected spans.
4. **Sidecar resource waste**: Each Envoy sidecar consumes ~50 MB RAM. For large clusters, consider ambient mode.
5. **Gateway API vs Istio API confusion**: Both work. Gateway API is the Kubernetes standard; Istio API (VirtualService/Gateway) offers more Istio-specific features. Pick one and be consistent.
6. **Waypoint not deployed for L7**: In ambient mode, L7 features (HTTP routing, L7 auth policies) require a waypoint proxy. Without it, only L4 mTLS and L4 auth work.
7. **istioctl version mismatch**: Always use `istioctl` version matching the installed Istio control plane version.

## Version-specific guidance

| Version | Reference | What's version-specific |
|---|---|---|
| 1.25 | `references/versions/1.25.md` | Ambient mesh stability, migration tooling |

## Reference Files

Read these for implementation depth:

- `references/architecture.md` -- istiod, Envoy, ztunnel, waypoint, HBONE, ambient vs sidecar internals. Read for architecture questions.
- `references/best-practices.md` -- Traffic management patterns, security policies, observability setup, ambient vs sidecar selection. Read for design questions.