Kubernetes

---
name: kubernetes
description: "Expert coverage of Kubernetes across all versions: control plane architecture, workload resources, scheduling, networking, storage, RBAC, autoscaling, and troubleshooting. Use for \"Kubernetes\", \"kubectl\", \"pod\", \"deployment\", \"service\", \"StatefulSet\", \"DaemonSet\", \"kube-apiserver\", \"etcd\", \"kubelet\". Do NOT use for GitOps continuous-delivery of manifests (ArgoCD sync waves, Flux reconciliation, app-of-apps) -- that's the `argocd` or `flux` skill (in the `devops` plugin)."
license: MIT
---

# Kubernetes

This skill covers Kubernetes across all supported versions (1.30 through 1.36), including:

- Control plane architecture (API server, etcd, scheduler, controller manager)
- Data plane components (kubelet, kube-proxy, container runtimes via CRI)
- Workload resources (Pods, Deployments, StatefulSets, DaemonSets, Jobs, CronJobs)
- Networking (Services, Ingress, Gateway API, NetworkPolicy, CNI)
- Storage (PV, PVC, StorageClass, CSI, VolumeAttributesClass)
- Security (RBAC, Pod Security Standards, ServiceAccounts, admission control)
- Scheduling (affinity, taints/tolerations, topology spread, priority/preemption)
- Autoscaling (HPA, VPA, KEDA, cluster autoscaler, Karpenter)
- Troubleshooting (kubectl debug, events, logs, pod lifecycle issues)

When a question is version-specific, read the matching file in the Version-specific guidance section below. When the version is unknown, provide general guidance and note where behavior differs across versions.

## How to Approach Tasks

1. **Classify** the request:
   - **Troubleshooting** -- Load `references/diagnostics.md` for kubectl commands, debug workflows, and event interpretation
   - **Architecture** -- Load `references/architecture.md` for control plane internals, packet flow, etcd, CRI
   - **Best practices** -- Load `references/best-practices.md` for resource management, security hardening, autoscaling design
   - **Workload design** -- Apply knowledge of Pods, Deployments, StatefulSets, Jobs below
   - **Networking** -- Cover Services, Ingress, Gateway API, NetworkPolicy, DNS
   - **Storage** -- Cover PV/PVC, StorageClass, CSI drivers, access modes

2. **Identify version** -- Determine which Kubernetes version the user runs. Features like sidecar containers (1.34 GA), user namespaces (1.35 beta), in-place pod resize (1.35 beta), nftables kube-proxy (1.33 beta) are version-gated. If version is unclear, ask.

3. **Load context** -- Read the relevant reference file for deep technical detail.

4. **Analyze** -- Apply Kubernetes-specific reasoning. Consider API group, resource version, feature gates.

5. **Recommend** -- Provide actionable guidance with YAML examples and kubectl commands.

6. **Verify** -- Suggest validation steps (kubectl describe, events, logs, dry-run).

## Control Plane Architecture

### kube-apiserver

The single entry point for all cluster operations. All clients (kubectl, controllers, operators) communicate exclusively with the API server via REST.

- Validates and persists resource definitions to etcd
- Enforces authentication (x509, OIDC, ServiceAccount tokens, webhook), authorization (RBAC, Node, Webhook), and admission control (mutating + validating webhooks, CEL-based policies)
- Horizontally scalable behind a load balancer
- API discovery: `kubectl api-resources`, `kubectl api-versions`

### etcd

Distributed key-value store. Source of truth for all cluster state.

- All objects stored as protobufs under `/registry/...` keys
- HA: 3 or 5 member cluster (Raft consensus). Even numbers provide no quorum benefit.
- Only the API server communicates with etcd directly
- Critical operations: `etcdctl snapshot save`, `etcdctl endpoint health`, defragmentation

### kube-scheduler

Watches for unscheduled Pods (`spec.nodeName == ""`) and assigns them:

1. **Filter**: eliminate nodes failing hard constraints (resources, taints, affinity, topology)
2. **Score**: rank remaining nodes (resource balance, spread, locality)
3. **Bind**: set `spec.nodeName` on the winning node

Extensible via the Scheduler Framework (QueueSort, PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind extension points).

### kube-controller-manager

Runs built-in controllers as goroutines: ReplicaSet, Deployment, StatefulSet, DaemonSet, Job, CronJob, Service, Namespace, Node, PV/PVC, ServiceAccount, EndpointSlice.

Each controller watches specific resources and reconciles desired vs actual state.

### cloud-controller-manager

Cloud-specific logic separated from core controllers: Node (validates node existence in cloud), Route (cloud network routes), Service (cloud load balancers).

## Data Plane

### kubelet

Agent on every node:
- Registers node with API server
- Watches for pods assigned to its node
- Manages container lifecycle via CRI (containerd or CRI-O)
- Runs probes (liveness, readiness, startup)
- Reports node and pod status
- Serves metrics at `/metrics/resource` (consumed by Metrics Server)
- Manages device plugins (GPUs, FPGAs)

### kube-proxy

Programs node networking rules for Service abstraction:

| Mode | Mechanism | Status |
|------|-----------|--------|
| iptables | DNAT rules per Service/Endpoint | Default (legacy) |
| ipvs | Linux IPVS for large-scale clusters | Stable |
| nftables | Modern kernel netfilter | Beta (1.33+) |

Increasingly replaced by eBPF-based CNI plugins (Cilium, Calico eBPF) that handle service load balancing in-kernel without kube-proxy.

### Container Runtime Interface (CRI)

gRPC interface between kubelet and container runtimes:
- **containerd**: default in most distributions, CRI plugin built-in
- **CRI-O**: lightweight, designed specifically for Kubernetes
- **Docker Engine**: via `cri-dockerd` shim (native CRI removed in K8s 1.24)

## Workload Resources

### Pods

Smallest deployable unit. One or more containers sharing network namespace and volumes.

Key concepts:
- **Init containers**: run sequentially before app containers, used for setup tasks
- **Sidecar containers** (stable in K8s 1.34): init containers with `restartPolicy: Always`, run alongside main containers for the pod's lifetime
- **QoS classes**: Guaranteed (requests==limits), Burstable (partial), BestEffort (none). Determines eviction priority.
- **Probes**: liveness (restart on failure), readiness (remove from endpoints), startup (delay liveness until ready)
- **Graceful termination**: SIGTERM → `terminationGracePeriodSeconds` → SIGKILL
- **Security context**: runAsNonRoot, readOnlyRootFilesystem, capabilities, seccomp profiles

### Deployments

Manage ReplicaSets for stateless workloads. Support rolling updates (`maxUnavailable`, `maxSurge`), rollbacks (`kubectl rollout undo`), and revision history.

### StatefulSets

For stateful workloads requiring:
- Stable network identities (pod-0, pod-1, ...)
- Ordered deployment and scaling
- Persistent volume per pod via `volumeClaimTemplates`
- Headless Service for DNS-based discovery

### DaemonSets

One pod per node (or per matching node). Common for log collectors, monitoring agents, CNI plugins.

### Jobs and CronJobs

Batch workloads. Jobs run to completion with configurable parallelism, completions, backoff, and TTL. CronJobs schedule Jobs on a cron expression with concurrency policies (Allow, Forbid, Replace).

## Networking

### Services

| Type | Behavior |
|------|----------|
| ClusterIP | Internal-only virtual IP |
| NodePort | ClusterIP + port on every node (30000-32767) |
| LoadBalancer | NodePort + cloud load balancer |
| ExternalName | CNAME alias to external DNS |
| Headless (`clusterIP: None`) | DNS returns pod IPs directly |

### Gateway API (v1.4, GA)

Replaces Ingress with role-oriented resource model:
- **GatewayClass**: infrastructure provider declares controller
- **Gateway**: infrastructure team configures listeners (ports, TLS)
- **HTTPRoute/GRPCRoute**: application teams define routing rules
- **TCPRoute/TLSRoute**: experimental L4 routing

Gateway API v1.4 adds BackendTLSPolicy for gateway-to-backend TLS and stable GRPCRoute.

### NetworkPolicy

Namespace-scoped firewall rules controlling pod-to-pod traffic. Requires a CNI that enforces them (Calico, Cilium, Weave). Default behavior: all traffic allowed. Adding any NetworkPolicy selecting a pod makes that pod default-deny for the specified policy types.

### DNS

CoreDNS provides cluster DNS. Service records: `<service>.<namespace>.svc.cluster.local`. Pod records available with headless services. Customizable via ConfigMap (`coredns` in `kube-system`).

## Storage

### PersistentVolumes (PV) and PersistentVolumeClaims (PVC)

PVs represent storage. PVCs request storage. StorageClasses enable dynamic provisioning.

Access modes: `ReadWriteOnce` (single node), `ReadOnlyMany`, `ReadWriteMany`, `ReadWriteOncePod` (single pod, 1.22+).

Reclaim policies: `Delete` (default for dynamic), `Retain` (manual cleanup).

**VolumeAttributesClass (VAC)**: GA in K8s 1.34. Allows modifying storage attributes (IOPS, throughput) without recreating PVCs.

### CSI (Container Storage Interface)

Plugin interface for storage providers. All major cloud and enterprise storage vendors provide CSI drivers. StorageClass references the CSI driver via `provisioner` field.

## Security

### RBAC

Four resources: Role (namespace-scoped), ClusterRole (cluster-scoped), RoleBinding, ClusterRoleBinding.

Rules specify apiGroups, resources, and verbs. Principle of least privilege: grant only necessary permissions. Audit with `kubectl auth can-i --list --as=user`.

### Pod Security Standards

Applied via namespace labels (`pod-security.kubernetes.io/enforce: restricted`):
- **Privileged**: no restrictions
- **Baseline**: blocks known privilege escalations
- **Restricted**: requires non-root, no privilege escalation, drop ALL capabilities, seccomp profile

### ServiceAccount Tokens

Bound ServiceAccount tokens (audience-scoped, time-limited) are default since K8s 1.24. Legacy non-expiring tokens no longer auto-created.

## Autoscaling

### HPA (Horizontal Pod Autoscaler)

Scales replica count based on CPU, memory, or custom/external metrics. Use `autoscaling/v2` API. Configure `behavior` for stabilization windows and scaling policies.

### VPA (Vertical Pod Autoscaler)

Adjusts resource requests/limits. Modes: Off (recommendations only), Initial (set at pod creation), Auto (evict and recreate with new resources). In-place pod resize (K8s 1.35 beta) will reduce VPA's need to restart pods.

**Important**: VPA and HPA must not scale the same metric simultaneously.

### Cluster-Level Autoscaling

- **Cluster Autoscaler**: adds/removes nodes when pods are unschedulable or nodes underutilized
- **Karpenter**: faster, more flexible node provisioning (selects optimal instance type per pod requirements). De facto standard on EKS; available on AKS.

## Scheduling Deep Dive

- **nodeSelector**: simple label-based node matching
- **nodeAffinity**: expressive node matching (In, NotIn, Exists, Gt, Lt operators)
- **podAffinity/podAntiAffinity**: co-locate or spread pods relative to other pods
- **Taints and Tolerations**: nodes repel pods unless pods explicitly tolerate the taint
- **Topology Spread Constraints**: distribute pods across failure domains with configurable maxSkew
- **PriorityClass and Preemption**: higher-priority pods can evict lower-priority pods

## Version-specific guidance

| Version | Reference | What's version-specific |
|---|---|---|
| 1.34 | `references/versions/1.34.md` | Sidecar containers GA, VolumeAttributesClass GA, Windows improvements, OIDC discovery |
| 1.35 | `references/versions/1.35.md` | User namespaces beta, in-place pod resize beta, DRA enhancements, cgroup v1 deprecation |

## Reference Files

Read these for implementation depth:

- `references/architecture.md` -- Control plane internals, etcd operations, API server flow, kubelet mechanics, CRI details, kube-proxy modes. Read for "how does X work" questions.
- `references/diagnostics.md` -- kubectl debug, describe, logs, events, resource troubleshooting, pod not starting flowcharts, node issues, network debugging. Read when troubleshooting.
- `references/best-practices.md` -- Resource management, RBAC design, NetworkPolicy strategy, Pod Security Standards adoption, HPA/VPA tuning, PDB configuration, upgrade procedures. Read for design and operations questions.

## Diagnostic Scripts

Ready-made kubectl bundles (read-only) in `scripts/`, numbered by investigation order.

- `scripts/01-pod-triage.sh` -- Not-Running/not-Ready pods, restart leaders, status-ladder guidance
- `scripts/02-node-triage.sh` -- NotReady nodes, pressure conditions, allocatable headroom
- `scripts/03-recent-events.sh` -- Warning events newest-first (the what-changed feed)
- `scripts/04-resource-pressure.sh` -- Top consumers plus containers without memory limits