Cert Manager

---
name: cert-manager
description: "Expert agent for cert-manager on Kubernetes (CNCF graduated). Covers Certificate resources, Issuers/ClusterIssuers (ACME, Vault, CA, Venafi, self-signed), DNS-01/HTTP-01 solvers, trust-manager for CA bundle distribution, and SPIFFE/CSI driver patterns. WHEN: \"cert-manager\", \"Kubernetes certificates\", \"ClusterIssuer\", \"Certificate resource\", \"cert-manager ACME\", \"cert-manager Vault\", \"trust-manager\", \"cert-manager CSI\", \"ACME solver\", \"certificate renewal Kubernetes\"."
license: MIT
---

# cert-manager

This skill covers cert-manager, the CNCF graduated Kubernetes add-on for certificate lifecycle management. It has deep knowledge of all issuer types, Certificate resources, renewal behavior, troubleshooting, and advanced patterns.

## How to Approach Tasks

1. **Classify the request**:
   - **Installation** — Helm, manifests, version selection
   - **Issuer configuration** — ACME, Vault, CA, Venafi, self-signed, external
   - **Certificate resources** — Certificate, CertificateRequest, Order, Challenge
   - **Solver configuration** — HTTP-01, DNS-01, TLS-ALPN-01
   - **trust-manager** — CA bundle distribution
   - **Troubleshooting** — Certificate not ready, challenge failing, renewal stuck
   - **CSI driver** — cert-manager-csi-driver patterns

2. **Identify issuer scope**: `Issuer` (namespace-scoped) vs. `ClusterIssuer` (cluster-wide).

3. **Identify Kubernetes environment**: Cloud (GKE, EKS, AKS) or on-prem (affects ingress class, DNS solver providers).

## Installation

```bash
# Install via Helm (recommended)
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --set crds.enabled=true \
    --version v1.17.0

# Verify
kubectl get pods -n cert-manager
kubectl get crds | grep cert-manager
```

### Verify Installation

```bash
# Create a test ClusterIssuer and Certificate
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.crds.yaml

# Quick test with self-signed certificate
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
  namespace: default
spec:
  secretName: test-cert-tls
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
  dnsNames:
    - example.com
EOF

kubectl describe certificate test-cert
```

---

## Issuer Types

### ACME (Let's Encrypt, ZeroSSL, etc.)

```yaml
# ClusterIssuer for Let's Encrypt production
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    
    # Store account key in this secret
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    
    solvers:
    # HTTP-01 solver for non-wildcard domains
    - http01:
        ingress:
          class: nginx  # or: ingressClassName: nginx
    
    # DNS-01 solver for wildcard domains
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
      selector:
        dnsZones:
          - "example.com"

---
# ClusterIssuer for Let's Encrypt staging (testing)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
```

### DNS-01 Solver Providers and Vault PKI Issuer

See `references/issuer-examples.md` for full DNS-01 solver configuration for Route 53, Azure DNS, Google Cloud DNS, and Cloudflare, plus the Vault PKI issuer spec and Vault-side setup (auth role, policy).

### CA Issuer (Internal CA)

```yaml
# Store CA cert + key in a Kubernetes Secret
kubectl create secret tls internal-ca-secret \
    --cert=ca.crt \
    --key=ca.key \
    -n cert-manager

---
# ClusterIssuer using that CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-secret
```

### Venafi Issuer

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: venafi-tpp-issuer
spec:
  venafi:
    zone: "\\VED\\Policy\\Kubernetes-TLS"
    tpp:
      url: https://tpp.example.com/vedsdk
      credentialsRef:
        name: venafi-tpp-credentials
      caBundle: <base64-ca>
```

### Self-Signed Issuer

Useful for bootstrapping (sign CA with self-signed, then use CA as issuer):

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

---
# Bootstrap a CA certificate using self-signed
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: "Internal CA"
  secretName: internal-ca-tls
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

---
# Use the bootstrapped CA as an issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-tls
```

---

## Certificate Resource

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: production
spec:
  # Kubernetes Secret where cert/key will be stored
  secretName: api-tls-secret
  
  # Certificate details
  commonName: api.example.com
  dnsNames:
    - api.example.com
    - api-v2.example.com
  ipAddresses:
    - 10.0.0.1
  
  # Validity period
  duration: 2160h      # 90 days
  renewBefore: 360h    # Renew 15 days before expiry
  
  # Key configuration
  privateKey:
    algorithm: ECDSA    # or RSA
    size: 256           # P-256 for ECDSA, 2048/4096 for RSA
    rotationPolicy: Always  # Always rotate key on renewal (vs. Never)
  
  # Certificate usage
  usages:
    - server auth
    - client auth       # Include if mTLS client cert too
  
  # Which issuer to use
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer  # or Issuer (namespace-scoped)
    group: cert-manager.io
  
  # Additional secret configuration
  secretTemplate:
    annotations:
      my-annotation: "value"
    labels:
      app: api
```

### Certificate Status and Conditions

```bash
# Check certificate status
kubectl describe certificate api-tls -n production
# Look for: Conditions (Ready=True), Events

# Check underlying resources
kubectl get certificaterequest -n production
kubectl get order -n production  # ACME only
kubectl get challenge -n production  # ACME only (during issuance)

# View the certificate content
kubectl get secret api-tls-secret -n production -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
```

### Annotating Ingress for Auto-Cert

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # Or for namespace-scoped Issuer:
    # cert-manager.io/issuer: "my-issuer"
spec:
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-secret  # cert-manager will create/manage this secret
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
```

---

## trust-manager

trust-manager distributes CA trust bundles (CA certificates) across namespaces as ConfigMaps or Secrets. Applications can mount the trust bundle to verify internal certificate chains.

```bash
# Install trust-manager
helm install trust-manager jetstack/trust-manager \
    --namespace cert-manager \
    --set app.trust.namespace=cert-manager
```

```yaml
# Bundle resource — defines a trust bundle
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
  # From a ConfigMap in the trust namespace
  - configMap:
      name: internal-ca-cert
      key: ca.crt
  # From a Secret (the cert part only, not key)
  - secret:
      name: internal-ca-tls
      key: tls.crt
  # Include public CA bundle from cert-manager's default bundle
  - useDefaultCAs: true
  
  target:
    # Sync to ConfigMap in all namespaces
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        bundle.cert-manager.io/inject: "true"
```

```yaml
# In application pod — mount the trust bundle
volumes:
- name: ca-bundle
  configMap:
    name: internal-ca-bundle
    items:
    - key: ca-bundle.crt
      path: ca-bundle.crt

containers:
- name: app
  volumeMounts:
  - name: ca-bundle
    mountPath: /etc/ssl/custom-certs
    readOnly: true
  env:
  - name: SSL_CERT_FILE
    value: /etc/ssl/custom-certs/ca-bundle.crt
```

---

## cert-manager CSI Driver

Mounts certificates directly as volumes (without creating Kubernetes Secrets) — better for high-churn, short-lived certificates since nothing is stored in etcd. See `references/csi-driver.md` for the Helm install and full pod CSI volume spec.

---

## Troubleshooting

### Certificate Not Ready

```bash
# Step 1: Check Certificate resource
kubectl describe certificate <name> -n <namespace>
# Look for: Reason, Message in Conditions

# Step 2: Check CertificateRequest
kubectl get certificaterequest -n <namespace>
kubectl describe certificaterequest <name> -n <namespace>

# Step 3: For ACME — check Order and Challenge
kubectl get order -n <namespace>
kubectl describe order <name> -n <namespace>
kubectl get challenge -n <namespace>
kubectl describe challenge <name> -n <namespace>

# Step 4: Check cert-manager controller logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100
```

### Common Issues

**HTTP-01 challenge failing**:
- Check Ingress controller is routing `/.well-known/acme-challenge/` to cert-manager's solver
- Check port 80 is accessible from Let's Encrypt servers (not just from within cluster)
- Check nginx ingress annotation `kubernetes.io/ingress.class` matches your issuer's `ingress.class`

**DNS-01 challenge failing**:
- Verify DNS API credentials are correct
- Check DNS propagation: `dig TXT _acme-challenge.example.com @8.8.8.8`
- Increase propagation wait time if needed

**Vault issuer failing**:
- Verify cert-manager service account has Vault auth role binding
- Check Vault PKI path in issuer spec matches actual mount path
- Check Vault is reachable from cert-manager pod: `kubectl exec -n cert-manager deploy/cert-manager -- curl https://vault.example.com`

**Certificate stuck in Pending**:
- Check if CertificateRequest exists: if not, cert-manager controller may not be running
- If CertificateRequest exists but Order not created: ACME server unreachable
- If Order exists but Challenge not completing: see HTTP-01/DNS-01 troubleshooting above

### Forcing Manual Renewal

```bash
# Delete the secret — cert-manager will re-issue
kubectl delete secret api-tls-secret -n production

# Or annotate Certificate to trigger renewal
kubectl annotate certificate api-tls -n production \
    cert-manager.io/issuer-name=letsencrypt-prod  # any annotation change triggers reconcile

# Or use cmctl (cert-manager CLI)
cmctl renew api-tls -n production
```

## Reference Files

Load these for deep knowledge on specific topics:

- `references/issuer-examples.md` — Full DNS-01 solver configs (Route 53, Azure DNS, Google Cloud DNS, Cloudflare) and Vault PKI issuer setup.
- `references/csi-driver.md` — cert-manager CSI driver install and full pod volume spec.