Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,116 @@
+# Gateway Lifecycle Troubleshooting
+
+Diagnostic flow when `hermes gateway restart` doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).
+
+## Symptom: `hermes gateway restart` fails with "linger is not enabled"
+
+```
+Cannot restart gateway as a service -- linger is not enabled.
+The gateway user service requires linger to function on headless servers.
+Run:  sudo loginctl enable-linger root
+```
+
+On containers, LXC, or VPS environments without a DBus session bus, this will fail with:
+
+```
+Failed to connect to system scope bus via local transport: Connection refused
+```
+
+On these systems the gateway may run via a **system-level** `hermes.service` unit (at `/etc/systemd/system/`), not a user-level one. The user-level service guard (`linger`) is a red herring -- the system-level unit uses `User=root` and the global `EnvironmentFile`, bypassing the DBus session issue entirely.
+
+## Diagnostic Steps
+
+### 1. Find what's running
+
+```bash
+# Check system-level service
+systemctl cat hermes.service          # shows the full unit
+systemctl status hermes.service       # active, failed, auto-restart loop?
+
+# Check for manually-started gateway processes
+ps aux | grep "hermes.*gateway" | grep -v grep
+```
+
+This reveals the critical fork: is the gateway running under systemd or as an orphan process?
+
+### 2. Categorize what you find
+
+| Situation | What to do |
+|-----------|-----------|
+| Gateway running under systemd (`hermes.service` active) | `systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level |
+| Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`) | `hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly |
+| `hermes.service` in `auto-restart` loop (exit-code 1) | Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process. |
+| No gateway at all | `systemctl start hermes.service` or `hermes gateway run` |
+
+### 3. The common cause: dual processes
+
+The gateway can be started in two ways:
+- **Manually:** `hermes gateway run` (or `hermes gateway run --replace`)
+- **As a systemd service:** `hermes.service` at `/etc/systemd/system/`
+
+If someone starts it manually (e.g. `hermes gateway run` in a tmux session), systemd's `hermes.service` will fail with "Gateway already running" and loop forever. The fix:
+
+```bash
+hermes gateway stop          # kills the manual process
+# systemd auto-restarts within 5s -- check:
+sleep 6 && systemctl is-active hermes.service
+```
+
+### 4. Stale systemd unit warning
+
+After restarting, check logs for:
+
+```
+Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.
+```
+
+This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:
+
+```bash
+hermes gateway install --force
+```
+
+This rewrites the unit with `TimeoutStopSec = drain_timeout + 30s`.
+
+### 5. User-level vs system-level units
+
+This environment has BOTH:
+- **System-level:** `/etc/systemd/system/hermes.service` -- runs the main Hermes gateway (default profile). `User=root`, uses `EnvironmentFile=/root/.hermes/.env`. This is the primary unit.
+- **User-level:** `~/.config/systemd/user/hermes-gateway-anita.service` -- runs the Anita profile gateway. Requires `loginctl enable-linger` (on this VPS, the system bus is unavailable so user-level units can't start).
+
+Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.
+
+### 6. Profile gateways
+
+The Anita profile runs as a separate systemd service. Check its status:
+
+```bash
+cat /proc/<pid>/cmdline | tr '\0' ' '   # tells you which profile is running
+ps aux | grep "profile anita"           # Anita's gateway process
+```
+
+Multiple gateway processes (one per profile) can run simultaneously without conflict.
+
+## Quick Reference
+
+```bash
+# See the unit definition
+systemctl cat hermes.service
+
+# Check gateway status
+hermes gateway status
+
+# Kill manual process, let systemd take over
+hermes gateway stop
+
+# Regenerate the systemd unit with correct timeouts
+hermes gateway install --force
+
+# Check latest logs
+journalctl -u hermes.service --no-pager -n 15
+
+# Check for any gateway-related processes
+ps aux | grep "hermes.*gateway" | grep -v grep
+# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
+# The first is manual, the second is often tmux/systemd-launched
+```