# Gateway Lifecycle Troubleshooting

Diagnostic flow when `hermes gateway restart` doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).

## Symptom: `hermes gateway restart` fails with "linger is not enabled"

```
Cannot restart gateway as a service -- linger is not enabled.
The gateway user service requires linger to function on headless servers.
Run:  sudo loginctl enable-linger root
```

On containers, LXC, or VPS environments without a DBus session bus, this will fail with:

```
Failed to connect to system scope bus via local transport: Connection refused
```

On these systems the gateway may run via a **system-level** `hermes.service` unit (at `/etc/systemd/system/`), not a user-level one. The user-level service guard (`linger`) is a red herring -- the system-level unit uses `User=root` and the global `EnvironmentFile`, bypassing the DBus session issue entirely.

## Diagnostic Steps

### 1. Find what's running

```bash
# Check system-level service
systemctl cat hermes.service          # shows the full unit
systemctl status hermes.service       # active, failed, auto-restart loop?

# Check for manually-started gateway processes
ps aux | grep "hermes.*gateway" | grep -v grep
```

This reveals the critical fork: is the gateway running under systemd or as an orphan process?

### 2. Categorize what you find

| Situation | What to do |
|-----------|-----------|
| Gateway running under systemd (`hermes.service` active) | `systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level |
| Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`) | `hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly |
| `hermes.service` in `auto-restart` loop (exit-code 1) | Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process. |
| No gateway at all | `systemctl start hermes.service` or `hermes gateway run` |

### 3. The common cause: dual processes

The gateway can be started in two ways:
- **Manually:** `hermes gateway run` (or `hermes gateway run --replace`)
- **As a systemd service:** `hermes.service` at `/etc/systemd/system/`

If someone starts it manually (e.g. `hermes gateway run` in a tmux session), systemd's `hermes.service` will fail with "Gateway already running" and loop forever. The fix:

```bash
hermes gateway stop          # kills the manual process
# systemd auto-restarts within 5s -- check:
sleep 6 && systemctl is-active hermes.service
```

### 4. Stale systemd unit warning

After restarting, check logs for:

```
Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.
```

This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:

```bash
hermes gateway install --force
```

This rewrites the unit with `TimeoutStopSec = drain_timeout + 30s`.

### 5. User-level vs system-level units

This environment has BOTH:
- **System-level:** `/etc/systemd/system/hermes.service` -- runs the main Hermes gateway (default profile). `User=root`, uses `EnvironmentFile=/root/.hermes/.env`. This is the primary unit.
- **User-level:** `~/.config/systemd/user/hermes-gateway-anita.service` -- runs the Anita profile gateway. Requires `loginctl enable-linger` (on this VPS, the system bus is unavailable so user-level units can't start).

Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.

### 6. Profile gateways

The Anita profile runs as a separate systemd service. Check its status:

```bash
cat /proc/<pid>/cmdline | tr '\0' ' '   # tells you which profile is running
ps aux | grep "profile anita"           # Anita's gateway process
```

Multiple gateway processes (one per profile) can run simultaneously without conflict.

## Quick Reference

```bash
# See the unit definition
systemctl cat hermes.service

# Check gateway status
hermes gateway status

# Kill manual process, let systemd take over
hermes gateway stop

# Regenerate the systemd unit with correct timeouts
hermes gateway install --force

# Check latest logs
journalctl -u hermes.service --no-pager -n 15

# Check for any gateway-related processes
ps aux | grep "hermes.*gateway" | grep -v grep
# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
# The first is manual, the second is often tmux/systemd-launched
```