Haproxy

---
name: haproxy
description: "Expert coverage of HAProxy across all versions: frontend/backend architecture, ACL-based routing, stick tables for rate limiting and session tracking, health checks, SSL offload, L4/L7 modes, runtime API for dynamic configuration, multi-threading, and Kubernetes Ingress Controller. Use for \"HAProxy\", \"haproxy.cfg\", \"frontend\", \"backend\", \"stick table\", \"ACL\", \"HAProxy runtime API\", \"HAProxy Ingress\". Do NOT use for cross-vendor comparison, platform selection, or category-wide architecture -- use the `load-balancing` skill."
license: MIT
---

# HAProxy

This skill covers HAProxy across all supported versions (2.8 LTS through 3.2 LTS and 3.3 current). Areas of expertise include:

- Frontend/backend/listen architecture with global and defaults sections
- ACL (Access Control List) system for complex routing and filtering
- Stick tables for stateful rate limiting, session tracking, and abuse detection
- Health checks (TCP, HTTP, external script-based)
- SSL/TLS offload and cipher management
- L4 (TCP mode) and L7 (HTTP mode) proxying
- Runtime API for dynamic configuration without reload
- Multi-threaded architecture (nbthread)
- Zero-downtime reloads (listener socket passing)
- Kubernetes Ingress Controller
- Logging architecture (async, no file I/O in data path)

When a question is version-specific, consult the matching `references/versions/` file. When the version is unknown, provide general guidance and note where behavior differs across versions.

## How to Approach Tasks

1. **Classify** the request:
   - **Troubleshooting** -- Check stats page, runtime API, logs, health check status
   - **Configuration** -- Load `references/best-practices.md` for SSL, health checks, rate limiting, tuning
   - **Architecture** -- Load `references/architecture.md` for frontend/backend model, ACLs, stick tables, runtime API
   - **Rate limiting** -- Apply stick table patterns below
   - **Kubernetes** -- Apply Ingress Controller guidance below

2. **Identify version** -- Determine HAProxy version (2.x or 3.x). If unclear, ask. Version matters for stick table features (3.0 sharding, 3.2 arrays), multi-threading behavior, and LTS status.

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply HAProxy-specific reasoning, not generic load balancer advice.

5. **Recommend** -- Provide actionable configuration snippets with explanations.

6. **Verify** -- Suggest validation steps (stats page, runtime API commands, log analysis).

## Core Architecture

### Process Model
- Single-process, multi-threaded (since 2.x)
- `nbthread` directive scales across CPU cores
- Event-driven: uses epoll (Linux) / kqueue (BSD) for non-blocking I/O
- No file I/O in data path: all logging is async (syslog UDP/TCP)
- Zero-downtime reload: new process inherits listener sockets from old process

### Configuration Structure
```
global          # Process-level: threading, SSL, logging, resource limits
defaults        # Default settings inherited by all frontends/backends
frontend        # Listener: accepts client connections
backend         # Server pool: forwards traffic to servers
listen          # Combined frontend + backend (shorthand)
```

### Frontend
Defines how HAProxy accepts incoming connections:
```haproxy
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/ssl/example.pem alpn h2,http/1.1
    
    http-request redirect scheme https unless { ssl_fc }
    
    # ACL-based routing
    acl is_api path_beg /api/
    use_backend api_backend if is_api
    default_backend web_backend
```

### Backend
Defines the server pool and load balancing behavior:
```haproxy
backend web_backend
    balance leastconn
    option httpchk GET /health HTTP/1.1\r\nHost:\ app.internal
    http-check expect status 200
    
    server web1 192.168.10.11:8080 check weight 10
    server web2 192.168.10.12:8080 check weight 10
    server web3 192.168.10.13:8080 check weight 5 backup
```

### Listen
Combined frontend+backend for simple deployments:
```haproxy
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:password
```

## ACL System

ACLs are named conditions used for routing, blocking, and traffic control:
```haproxy
frontend http_front
    # Define ACLs
    acl is_api          path_beg /api/
    acl is_admin        path_beg /admin/
    acl is_mobile       hdr_sub(User-Agent) -i Mobile
    acl internal_src    src 10.0.0.0/8 192.168.0.0/16
    acl has_auth        req.hdr(Authorization) -m found
    acl is_post         method POST
    acl is_large_body   req.body_len gt 1048576
    
    # Route based on ACLs
    use_backend api_backend     if is_api
    use_backend admin_backend   if is_admin internal_src
    http-request deny           if is_admin !internal_src
    use_backend upload_backend  if is_post is_large_body
    default_backend web_backend
```

**ACL matching methods**:
- `path_beg`, `path_end`, `path_reg`: URL path matching
- `hdr()`, `hdr_beg()`, `hdr_sub()`, `hdr_reg()`: Header matching
- `src`: Source IP matching
- `ssl_fc`: SSL/TLS connection (boolean)
- `method`: HTTP method matching
- `-m found`: Check if header/value exists
- `-i`: Case-insensitive matching

## Stick Tables

Stick tables are in-memory data stores for stateful tracking and rate limiting.

### Rate Limiting with Stick Tables
```haproxy
frontend http_front
    # Define stick table: track per-IP request rate
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    
    # Track client IP
    http-request track-sc0 src
    
    # Deny if more than 100 requests in 10 seconds
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
```

### Session Tracking
```haproxy
backend app_backend
    stick-table type ip size 100k expire 10m store conn_cur,http_req_rate(1m),http_err_rate(1m)
    
    stick on src          # persist client IP to same server
    
    server app1 192.168.10.11:8080 check
    server app2 192.168.10.12:8080 check
```

### Stick Table Data Types
- `conn_cur`: Current concurrent connections
- `conn_rate(period)`: Connection rate
- `http_req_rate(period)`: HTTP request rate
- `http_err_rate(period)`: HTTP error rate (4xx/5xx)
- `gpc0`, `gpc1`: General Purpose Counters
- `gpt0`: General Purpose Tag
- `bytes_in_rate`, `bytes_out_rate`: Bandwidth tracking

## Health Checks

### HTTP Health Check
```haproxy
backend app_backend
    option httpchk GET /health HTTP/1.1\r\nHost:\ app.internal
    http-check expect status 200
    http-check expect string "healthy"
    
    server app1 192.168.10.11:8080 check inter 2s rise 2 fall 3
```

**Parameters**:
- `inter`: Check interval (default 2s)
- `rise`: Consecutive successes to mark UP (default 2)
- `fall`: Consecutive failures to mark DOWN (default 3)
- `fastinter`: Interval during transition (faster detection)
- `downinter`: Interval when server is down (slower to reduce load)

### TCP Health Check
```haproxy
backend db_backend
    mode tcp
    option tcp-check
    
    server db1 192.168.10.20:5432 check inter 5s
```

### External Script Health Check
```haproxy
backend custom_backend
    option external-check
    external-check command /usr/local/bin/check_app.sh
    
    server app1 192.168.10.11:8080 check inter 10s
```

## SSL/TLS Offload

```haproxy
global
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
    ssl-default-bind-options ssl-min-ver TLSv1.2
    tune.ssl.default-dh-param 2048

frontend https_front
    bind *:443 ssl crt /etc/ssl/certs/bundle.pem alpn h2,http/1.1
    
    # Forward proto and client cert info
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-SSL-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    
    default_backend app_backend
```

### SNI-Based Routing
```haproxy
frontend https_front
    bind *:443 ssl crt /etc/ssl/certs/ strict-sni
    
    # Route based on SNI (TLS hostname)
    use_backend api_backend if { ssl_fc_sni api.example.com }
    use_backend web_backend if { ssl_fc_sni www.example.com }
    default_backend default_web
```

## L4 (TCP) vs L7 (HTTP) Mode

### TCP Mode (L4)
```haproxy
frontend db_front
    mode tcp
    bind *:5432
    default_backend db_backend

backend db_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server db1 192.168.10.20:5432 check
    server db2 192.168.10.21:5432 check
```

Use for: databases (PostgreSQL, MySQL, Redis), SMTP, custom TCP protocols, any non-HTTP traffic.

### HTTP Mode (L7)
```haproxy
frontend web_front
    mode http
    bind *:80
    default_backend web_backend

backend web_backend
    mode http
    balance leastconn
    option httpchk
    server web1 192.168.10.11:8080 check
```

Use for: web applications, APIs, any HTTP/HTTPS traffic requiring content-based decisions.

## Runtime API

The Runtime API allows dynamic reconfiguration without reload:

### Enable Runtime API
```haproxy
global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
```

### Common Runtime Commands
```bash
# System information
echo "show info" | socat stdio /run/haproxy/admin.sock

# Show all server/backend statistics
echo "show stat" | socat stdio /run/haproxy/admin.sock

# Disable server (drain connections)
echo "disable server app_backend/app1" | socat stdio /run/haproxy/admin.sock

# Enable server
echo "enable server app_backend/app1" | socat stdio /run/haproxy/admin.sock

# Set server weight dynamically
echo "set weight app_backend/app1 50" | socat stdio /run/haproxy/admin.sock

# Show stick table contents
echo "show table STICK_TABLE_NAME" | socat stdio /run/haproxy/admin.sock

# Clear stick table
echo "clear table STICK_TABLE_NAME" | socat stdio /run/haproxy/admin.sock

# Show backend state
echo "show backend" | socat stdio /run/haproxy/admin.sock

# Show errors
echo "show errors" | socat stdio /run/haproxy/admin.sock
```

## Load Balancing Algorithms

| Algorithm | Directive | Description |
|---|---|---|
| Round Robin | `balance roundrobin` | Sequential distribution with weights |
| Static Round Robin | `balance static-rr` | No dynamic weight changes; faster |
| Least Connections | `balance leastconn` | Fewest active connections |
| Source | `balance source` | Client IP hash (session affinity) |
| URI | `balance uri` | Hash of URI (cache optimization) |
| Header | `balance hdr(name)` | Hash of HTTP header value |
| Random | `balance random` | Random; `random(2)` for P2C |
| First | `balance first` | Fill first server before moving to next |
| rdp-cookie | `balance rdp-cookie(name)` | RDP session persistence |

## Kubernetes Ingress Controller

HAProxy Technologies provides an official Kubernetes Ingress Controller:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    haproxy.org/load-balance: "leastconn"
    haproxy.org/timeout-connect: "5s"
    haproxy.org/rate-limit-requests: "100"
    haproxy.org/rate-limit-period: "1m"
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80
```

Features:
- Standard Ingress resources with HAProxy-specific annotations
- Native stick table support for session persistence
- Dynamic reconfiguration via ConfigMap + CRD
- Load balancing algorithms: roundrobin, leastconn, source, uri, hdr

## Common Pitfalls

1. **Mode mismatch** -- Setting `mode tcp` on a frontend but `mode http` on the backend (or vice versa) causes silent failures. Always match modes between frontend and backend.

2. **Forgetting `check` on server lines** -- Without the `check` keyword, no health checking is performed for that server. Always add `check` to enable health monitoring.

3. **Stick table memory sizing** -- `stick-table type ip size 100k` allocates space for 100,000 entries. If the table fills up, new entries are rejected. Size based on expected unique client count.

4. **ACL evaluation order** -- `use_backend` rules are evaluated top-down. If a broad ACL is listed before a specific one, the specific rule never matches. Order from most specific to least specific.

5. **Runtime API changes not persistent** -- Changes made via the runtime API (disable server, set weight) are lost on reload. For permanent changes, update haproxy.cfg.

6. **SSL certificate ordering in bundle** -- HAProxy requires the certificate bundle in order: server cert, intermediate(s), root. Incorrect ordering causes validation failures.

7. **Not using `option httplog`** -- Without `option httplog`, HAProxy uses `tcplog` format which lacks HTTP-specific fields (URL, status code, response time). Always enable `option httplog` for HTTP mode.

8. **Zero-downtime reload failure** -- `expose-fd listeners` must be set on the stats socket for seamless listener socket transfer during reload. Without it, reload causes brief connection drops.

## Version-Specific Guidance

| Version | Reference | What's version-specific |
|---|---|---|
| 3.2 | `references/versions/3.2.md` | HAProxy 3.2 LTS, stick table peer sync thread, array data types, enhanced show events |

## Reference Files

Load these when you need deep knowledge:

- `references/architecture.md` -- Frontend/backend model, ACL system, stick table internals, runtime API mechanics, multi-threading. Read for "how does X work" questions.
- `references/best-practices.md` -- SSL offload patterns, health check design, rate limiting strategies, K8s integration, performance tuning. Read for design and configuration questions.

## Diagnostic Scripts

Ready-made stats-socket and config-check scripts (read-only) in `scripts/`.

- `scripts/01-runtime-stats.sh` -- Down servers with check reasons, queueing, error rates
- `scripts/02-config-check.sh` -- Config validation gate before reload (and the reload-vs-restart rule)