Service Mesh

---
name: service-mesh
description: "Compares service mesh technologies -- Istio, Linkerd, and Consul Connect -- and helps decide whether a mesh is needed at all. Use for \"service mesh\", \"Istio vs Linkerd\", \"which service mesh\", \"do I need a service mesh\", \"sidecar proxy\", \"ambient mesh\", \"mTLS\", \"service-to-service\", \"mesh comparison\", \"zero trust networking\". Do NOT use for implementation questions about a specific mesh -- use the `istio`, `linkerd`, or `consul` skill."
license: MIT
---

# Service Mesh Overview

This skill covers service mesh technology selection. Use it to decide if a mesh is needed and compare architectures, then read a technology-specific skill for deep implementation questions.

## When to Use This Skill vs. a Technology Skill

**Use this skill when:**
- Deciding whether a service mesh is needed
- Comparing Istio, Linkerd, and Consul Connect
- Architecture and design questions that span mesh technologies
- Understanding service mesh concepts (mTLS, sidecar, data plane, control plane)

**Read a technology skill when:**
- Istio-specific: VirtualService, ambient mesh, ztunnel, Envoy config --> the `istio` skill
- Linkerd-specific: linkerd2-proxy, ServiceProfile, multi-cluster --> the `linkerd` skill
- Consul-specific: Intentions, ServiceRouter, mesh gateways --> the `consul` skill

## How to Approach Tasks

1. **Determine need** -- Does this workload actually need a service mesh? Many applications do not.
2. **Gather context** -- Platform (Kubernetes-only? VMs? hybrid?), scale, team expertise, existing infrastructure, compliance requirements
3. **Load** `references/concepts.md` for service mesh fundamentals if the question involves core concepts
4. **Compare** -- Apply decision framework below with concrete trade-offs
5. **Recommend** -- Specific mesh with rationale, and point to the appropriate technology skill

## Do You Need a Service Mesh?

### Yes, Consider a Service Mesh When

- **mTLS everywhere**: Regulatory or compliance requirement for encrypted pod-to-pod traffic (PCI-DSS, HIPAA, SOC2)
- **Zero trust networking**: Need identity-based authorization between services, not just network-level controls
- **Complex traffic management**: Canary deployments, A/B testing, traffic mirroring, circuit breaking across many services
- **Observability gaps**: Need per-request metrics, distributed tracing, and service topology without instrumenting every service
- **Multi-cluster/multi-cloud**: Services span multiple clusters that need unified networking and security

### No, Skip the Service Mesh When

- **Small number of services** (< 10): Network policies + application-level TLS is simpler
- **Single cluster, trusted network**: If all services are in one cluster and you trust the network, a mesh adds overhead without clear benefit
- **Team lacks Kubernetes expertise**: A mesh adds significant operational complexity on top of Kubernetes
- **Performance-critical path**: Every mesh adds latency (1-5ms per hop). For ultra-low-latency requirements, evaluate carefully
- **Batch/data processing**: Jobs that don't communicate service-to-service gain nothing from a mesh

### Alternatives to a Full Service Mesh

| Need | Alternative |
|---|---|
| mTLS only | cert-manager + application TLS, or SPIFFE/SPIRE |
| Traffic splitting | Kubernetes Gateway API + ingress controller |
| Observability | OpenTelemetry + instrumentation library |
| Network policy | Cilium NetworkPolicy (L3/L4/L7) |
| Service discovery | CoreDNS (built into Kubernetes) |

## Service Mesh Comparison

### Architecture Comparison

```
Istio (Sidecar mode):
  istiod (control plane) --> Envoy sidecar per pod (data plane)
  All L4+L7 in every sidecar

Istio (Ambient mode, GA in 1.24):
  istiod --> ztunnel per node (L4: mTLS, auth) + waypoint per namespace (L7: routing, optional)
  Sidecar-less, lower overhead

Linkerd:
  Control plane (destination, identity, proxy-injector) --> linkerd2-proxy per pod (Rust, minimal)
  Opinionated, zero-config mTLS

Consul Connect:
  Consul servers (raft cluster) --> Consul Dataplane + Envoy per service
  Multi-platform (K8s + VMs + bare metal)
```

### Decision Matrix

| Requirement | Istio | Linkerd | Consul Connect |
|---|---|---|---|
| Kubernetes-only | Best (ambient/sidecar) | Best (simplest) | Good |
| Multi-platform (K8s + VMs) | No | No | Best |
| Simplicity / fast adoption | Medium | Best | Medium |
| Traffic management richness | Best | Basic | Good |
| Resource overhead | Medium (ambient: low) | Lowest | Medium |
| Control plane memory | ~600 MB (ambient) / ~1-2 GB (sidecar) | ~200-300 MB | ~500 MB (servers) |
| Proxy RAM per pod | ~50 MB (Envoy) / shared ztunnel | ~20-30 MB (Rust) | ~50 MB (Envoy) |
| mTLS setup complexity | PeerAuthentication CRD | Zero config (automatic) | Auto with ACLs |
| L7 routing | VirtualService + Gateway API | ServiceProfile / SMI | ServiceRouter / ServiceSplitter |
| Multi-cluster | Alpha (ambient 1.27) | Gateway mirroring (stable) | Native WAN federation (best) |
| JWT authentication | Built-in (RequestAuthentication) | External | JWT filter |
| Post-quantum crypto | No | Yes (Linkerd 2.19+, ML-KEM-768) | No |
| Ecosystem / community | Largest (CNCF graduated) | Growing (CNCF graduated) | HashiCorp ecosystem |
| OpenShift integration | Red Hat OSSM | Community support | Certified |
| UI dashboard | Kiali (topology, health) | Linkerd Viz (built-in) | Consul UI |
| p99 latency overhead | Higher | Lowest (163ms less at 2k RPS) | Medium |

### Selection Recommendations

| Your Situation | Recommended Mesh | Rationale |
|---|---|---|
| Kubernetes-only, want simplicity | **Linkerd** | Zero-config mTLS, lowest overhead, fastest adoption |
| Kubernetes-only, need rich traffic management | **Istio Ambient** | VirtualService, Gateway API, waypoint proxies for L7 |
| Migrating from Istio sidecar | **Istio Ambient** | Incremental namespace-by-namespace migration |
| Multi-platform (K8s + VMs + bare metal) | **Consul Connect** | Only mesh that natively supports non-K8s workloads |
| HashiCorp stack (Vault, Nomad, Terraform) | **Consul Connect** | Deep integration with HashiCorp ecosystem |
| OpenShift 4.x | **Istio** (via Red Hat OSSM) | Officially supported by Red Hat |
| Post-quantum security requirement | **Linkerd 2.19+** | ML-KEM-768 hybrid key exchange in mTLS |
| Need largest ecosystem and tooling | **Istio** | Kiali, Jaeger, extensive documentation, large community |
| Performance-critical, minimal latency | **Linkerd** | Rust proxy, 4-6x less control plane memory |

## Migration Between Meshes

### Common Migration Patterns

**Istio Sidecar to Istio Ambient:**
1. Install ambient profile alongside existing sidecar
2. Migrate namespace-by-namespace: `kubectl label namespace production istio.io/dataplane-mode=ambient`
3. Remove sidecar injection label
4. Deploy waypoint proxies where L7 features are needed
5. No application changes required

**No Mesh to Linkerd:**
1. Install Linkerd CRDs and control plane
2. Annotate namespaces: `linkerd.io/inject=enabled`
3. Restart deployments to inject proxy
4. mTLS is automatic -- no additional configuration

**Docker Compose to Consul Connect:**
1. Deploy Consul servers (raft cluster)
2. Register services with Consul catalog
3. Install Consul Dataplane + Envoy sidecar per service
4. Define intentions for access control
5. Enable transparent proxy for automatic traffic capture

## Observability Stack Integration

All major meshes integrate with the same observability tools:

| Layer | Tools |
|---|---|
| Metrics | Prometheus + Grafana |
| Tracing | Jaeger, Zipkin, Tempo (via OpenTelemetry) |
| Logging | Fluentd/Fluent Bit, Loki |
| Topology | Kiali (Istio), Linkerd Viz, Consul UI |

**Important**: Applications must propagate trace context headers for distributed tracing to work. The mesh proxies generate their own spans but cannot correlate them without application-level header propagation.

## Common Pitfalls

1. **Adopting a mesh too early**: Adding a mesh to a small number of services creates operational overhead with minimal benefit. Start with network policies and application-level TLS.
2. **Ignoring resource overhead**: Every sidecar proxy consumes CPU and memory. At scale (1000+ pods), the aggregate overhead is significant. Evaluate ambient/sidecar-less options.
3. **mTLS PERMISSIVE mode in production**: Permissive mode accepts both encrypted and plaintext traffic. Use STRICT mode after migration is complete.
4. **Not propagating trace headers**: The mesh generates proxy-level spans, but without application header propagation, traces are disconnected fragments.
5. **Multi-cluster without planning**: Cross-cluster mesh networking is complex. Start with single-cluster and expand deliberately.
6. **Mixing meshes**: Running multiple meshes in the same cluster causes conflicts (iptables rules, port conflicts). Choose one.

## Related Skills

Read these for deep implementation expertise:

- `istio` -- Istio (sidecar and ambient mode, VirtualService, Gateway, security policies; version-specific guidance for 1.25 in `references/versions/1.25.md`)
- `linkerd` -- Linkerd (linkerd2-proxy, ServiceProfile, multi-cluster, post-quantum)
- `consul` -- Consul Connect (Dataplane, intentions, ServiceRouter, mesh gateways)

## Reference Files

- `references/concepts.md` -- Service mesh fundamentals (sidecar pattern, data plane, control plane, mTLS, traffic management, observability). Read for "what is X" conceptual questions.