Gateway Lifecycle Troubleshooting

Diagnostic flow when hermes gateway restart doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).

Symptom: `hermes gateway restart` fails with "linger is not enabled"

Cannot restart gateway as a service -- linger is not enabled.
The gateway user service requires linger to function on headless servers.
Run:  sudo loginctl enable-linger root

On containers, LXC, or VPS environments without a DBus session bus, this will fail with:

Failed to connect to system scope bus via local transport: Connection refused

On these systems the gateway may run via a system-level hermes.service unit (at /etc/systemd/system/), not a user-level one. The user-level service guard (linger) is a red herring -- the system-level unit uses User=root and the global EnvironmentFile, bypassing the DBus session issue entirely.

Diagnostic Steps

1. Find what's running

# Check system-level service
systemctl cat hermes.service          # shows the full unit
systemctl status hermes.service       # active, failed, auto-restart loop?

# Check for manually-started gateway processes
ps aux | grep "hermes.*gateway" | grep -v grep

This reveals the critical fork: is the gateway running under systemd or as an orphan process?

2. Categorize what you find

Situation	What to do
Gateway running under systemd (`hermes.service` active)	`systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level
Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`)	`hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly
`hermes.service` in `auto-restart` loop (exit-code 1)	Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process.
No gateway at all	`systemctl start hermes.service` or `hermes gateway run`

3. The common cause: dual processes

The gateway can be started in two ways:

Manually: hermes gateway run (or hermes gateway run --replace)
As a systemd service: hermes.service at /etc/systemd/system/

If someone starts it manually (e.g. hermes gateway run in a tmux session), systemd's hermes.service will fail with "Gateway already running" and loop forever. The fix:

hermes gateway stop          # kills the manual process
# systemd auto-restarts within 5s -- check:
sleep 6 && systemctl is-active hermes.service

4. Stale systemd unit warning

After restarting, check logs for:

Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.

This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:

hermes gateway install --force

This rewrites the unit with TimeoutStopSec = drain_timeout + 30s.

5. User-level vs system-level units

This environment has BOTH:

System-level: /etc/systemd/system/hermes.service -- runs the main Hermes gateway (default profile). User=root, uses EnvironmentFile=/root/.hermes/.env. This is the primary unit.
User-level: ~/.config/systemd/user/hermes-gateway-anita.service -- runs the Anita profile gateway. Requires loginctl enable-linger (on this VPS, the system bus is unavailable so user-level units can't start).

Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.

6. Profile gateways

The Anita profile runs as a separate systemd service. Check its status:

cat /proc/<pid>/cmdline | tr '\0' ' '   # tells you which profile is running
ps aux | grep "profile anita"           # Anita's gateway process

Multiple gateway processes (one per profile) can run simultaneously without conflict.

Quick Reference

# See the unit definition
systemctl cat hermes.service

# Check gateway status
hermes gateway status

# Kill manual process, let systemd take over
hermes gateway stop

# Regenerate the systemd unit with correct timeouts
hermes gateway install --force

# Check latest logs
journalctl -u hermes.service --no-pager -n 15

# Check for any gateway-related processes
ps aux | grep "hermes.*gateway" | grep -v grep
# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
# The first is manual, the second is often tmux/systemd-launched

4.5 KiB Raw Permalink Blame History