117 lines
4.5 KiB
Markdown
117 lines
4.5 KiB
Markdown
# Gateway Lifecycle Troubleshooting
|
|
|
|
Diagnostic flow when `hermes gateway restart` doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).
|
|
|
|
## Symptom: `hermes gateway restart` fails with "linger is not enabled"
|
|
|
|
```
|
|
Cannot restart gateway as a service -- linger is not enabled.
|
|
The gateway user service requires linger to function on headless servers.
|
|
Run: sudo loginctl enable-linger root
|
|
```
|
|
|
|
On containers, LXC, or VPS environments without a DBus session bus, this will fail with:
|
|
|
|
```
|
|
Failed to connect to system scope bus via local transport: Connection refused
|
|
```
|
|
|
|
On these systems the gateway may run via a **system-level** `hermes.service` unit (at `/etc/systemd/system/`), not a user-level one. The user-level service guard (`linger`) is a red herring -- the system-level unit uses `User=root` and the global `EnvironmentFile`, bypassing the DBus session issue entirely.
|
|
|
|
## Diagnostic Steps
|
|
|
|
### 1. Find what's running
|
|
|
|
```bash
|
|
# Check system-level service
|
|
systemctl cat hermes.service # shows the full unit
|
|
systemctl status hermes.service # active, failed, auto-restart loop?
|
|
|
|
# Check for manually-started gateway processes
|
|
ps aux | grep "hermes.*gateway" | grep -v grep
|
|
```
|
|
|
|
This reveals the critical fork: is the gateway running under systemd or as an orphan process?
|
|
|
|
### 2. Categorize what you find
|
|
|
|
| Situation | What to do |
|
|
|-----------|-----------|
|
|
| Gateway running under systemd (`hermes.service` active) | `systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level |
|
|
| Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`) | `hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly |
|
|
| `hermes.service` in `auto-restart` loop (exit-code 1) | Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process. |
|
|
| No gateway at all | `systemctl start hermes.service` or `hermes gateway run` |
|
|
|
|
### 3. The common cause: dual processes
|
|
|
|
The gateway can be started in two ways:
|
|
- **Manually:** `hermes gateway run` (or `hermes gateway run --replace`)
|
|
- **As a systemd service:** `hermes.service` at `/etc/systemd/system/`
|
|
|
|
If someone starts it manually (e.g. `hermes gateway run` in a tmux session), systemd's `hermes.service` will fail with "Gateway already running" and loop forever. The fix:
|
|
|
|
```bash
|
|
hermes gateway stop # kills the manual process
|
|
# systemd auto-restarts within 5s -- check:
|
|
sleep 6 && systemctl is-active hermes.service
|
|
```
|
|
|
|
### 4. Stale systemd unit warning
|
|
|
|
After restarting, check logs for:
|
|
|
|
```
|
|
Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.
|
|
```
|
|
|
|
This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:
|
|
|
|
```bash
|
|
hermes gateway install --force
|
|
```
|
|
|
|
This rewrites the unit with `TimeoutStopSec = drain_timeout + 30s`.
|
|
|
|
### 5. User-level vs system-level units
|
|
|
|
This environment has BOTH:
|
|
- **System-level:** `/etc/systemd/system/hermes.service` -- runs the main Hermes gateway (default profile). `User=root`, uses `EnvironmentFile=/root/.hermes/.env`. This is the primary unit.
|
|
- **User-level:** `~/.config/systemd/user/hermes-gateway-anita.service` -- runs the Anita profile gateway. Requires `loginctl enable-linger` (on this VPS, the system bus is unavailable so user-level units can't start).
|
|
|
|
Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.
|
|
|
|
### 6. Profile gateways
|
|
|
|
The Anita profile runs as a separate systemd service. Check its status:
|
|
|
|
```bash
|
|
cat /proc/<pid>/cmdline | tr '\0' ' ' # tells you which profile is running
|
|
ps aux | grep "profile anita" # Anita's gateway process
|
|
```
|
|
|
|
Multiple gateway processes (one per profile) can run simultaneously without conflict.
|
|
|
|
## Quick Reference
|
|
|
|
```bash
|
|
# See the unit definition
|
|
systemctl cat hermes.service
|
|
|
|
# Check gateway status
|
|
hermes gateway status
|
|
|
|
# Kill manual process, let systemd take over
|
|
hermes gateway stop
|
|
|
|
# Regenerate the systemd unit with correct timeouts
|
|
hermes gateway install --force
|
|
|
|
# Check latest logs
|
|
journalctl -u hermes.service --no-pager -n 15
|
|
|
|
# Check for any gateway-related processes
|
|
ps aux | grep "hermes.*gateway" | grep -v grep
|
|
# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
|
|
# The first is manual, the second is often tmux/systemd-launched
|
|
```
|