hermes-skills/skills/devops/infrastructure-automation/references/caddy-proxy-port-mismatch.md

Caddy Proxy Port Mismatch — Deployment Tracking Pitfall

Problem

A Caddy reverse proxy entry points to a different port than the service is actually running on. The service has been running for days (sometimes via systemd), but the site returns 502 or ERR_CONNECTION_REFUSED because Caddy is proxying to a dead port.

Root Cause

When a service is deployed, three things must stay in sync:

1. The systemd service / Docker container port (e.g., --port 8083)
2. The systemd unit file ExecStart line (e.g., uvicorn server:app --host 0.0.0.0 --port 8083)
3. The Caddyfile reverse_proxy target (e.g., reverse_proxy 127.0.0.1:8083)

If one of these is updated without updating the others, the port drifts. The most common failure: Caddy proxies to port A (from an old config or manual edit), but the actual service runs on port B.

Example (shark.iamgmb.com, Jul 12, 2026)

• Caddy had: reverse_proxy 127.0.0.1:8081
• Actual backend: systemd service → port 8083 (confirmed by journalctl showing "Uvicorn running on http://0.0.0.0:8083")
• Result: shark.iamgmb.com returned 502 for 23+ hours despite the backend being active

Diagnosis

# Check what Caddy proxies to
grep -A2 'shark\.iamgmb\.com' /etc/caddy/Caddyfile

# Check what port the service actually binds
journalctl -u shark-game.service --no-pager | grep -i 'running on\|listening on\|port'

# Check if anything responds on Caddy's target port
ss -tlnp | grep ':8081'    # empty = no one listening
ss -tlnp | grep ':8083'    # should show python3

Fix

sed -i 's/reverse_proxy 127.0.0.1:8081/reverse_proxy 127.0.0.1:8083/' /etc/caddy/Caddyfile
caddy fmt --overwrite /etc/caddy/Caddyfile
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

Prevention

When deploying or updating any service behind Caddy:

1. Verify the port in the systemd unit file matches ExecStart
2. Verify the port in Caddyfile matches the service port
3. After any change, confirm with ss -tlnp | grep ':PORT' that the service is listening
4. Use curl -sS -o /dev/null -w '%{http_code}' https://domain.com/ to confirm the site responds