# Hetzner Server Inventory Compilation

When asked to document all Hetzner servers, services, and running applications:

## 1. Query the Hetzner API for server list

```python
import urllib.request, json
token = open('/root/.hermes/scripts/.hetzner_token').read().strip()
headers = {'Authorization': f'Bearer {token}'}
req = urllib.request.Request('https://api.hetzner.cloud/v1/servers', headers=headers)
data = json.loads(urllib.request.urlopen(req, timeout=10).read())

for s in data.get('servers', []):
    print(f'{s["name"]} — {s.get("public_net",{}).get("ipv4",{}).get("ip","?")} — {s.get("status")}')
```

## 2. Cross-reference with existing documentation

Check these sources in order:
- `/root/.hermes/skills/devops/infrastructure-automation/references/server-inventory-snapshot.md` — shows app-to-server mapping
- `/root/.hermes/skills/devops/infrastructure-automation/references/disaster-recovery-plan.md` — shows DR architecture
- Scripts in `/root/.hermes/scripts/` — function calls, IP references, API keys (e.g. `snapshot-hetzner.py`)
- Memory store (`memory` tool) and `facts_store` for past user statements

## 3. Mark verified vs assumed

| Marking | Meaning |
|---|---|
| 🟢 Verified | Confirmed by SSH or API response |
| 🟡 Assumed | From old inventory doc, needs SSH verification |
| 🔴 Unknown | No data available |

**SSH verification:** If you can SSH in with the wisp_rsa key, run `docker ps`, `systemctl list-units --type=service --state=running`, and `hostnamectl status` to confirm. If no SSH access exists, document it as a gap.

## 4. Document the access map

Not every server has the same SSH keys deployed. Track which servers are accessible with which keys:

| Server | SSH User | Key | Access |
|---|---|---|---|
| app1 (netcup) | root | wisp_rsa | ✅ |
| app1-bu (Hetzner) | root | wisp_rsa | ✅ |
| ai.itpropartner.com | ippadmin | ? | ❌ |

## 5. S3 backup

Upload the finished inventory to `s3://hermes-vps-backups/standby/server-inventory.md` for DR access.

## Pitfalls

- **Hetzner API hostname vs OS hostname** — `GET /v1/servers/{id}` returns the cloud provider's label, which may differ from the OS hostname. Both are valid — note which you're reporting.
- **Missing SSH key for most servers** — The wisp_rsa key was only deployed on app1 (netcup) and app1-bu (standby). All other Hetzner boxes need the SSH key injected via rescue mode (see `infrastructure-automation` skill's Hetzner rescue section) or the user's own key.
- **Old inventory may be stale** — apps may have been consolidated between servers since the document was written. Mark "needs SSH verification" rather than stating as fact.
- **The `data.get('datacenter',{})` may return empty** — The Hetzner API response has a `datacenter` field with a `location` sub-object (`city`, `country`). Not all API shapes include these. Handle gracefully.
- **Don't fabricate service details** — If you can't SSH in to verify what's running, say "from old inventory — needs verification" rather than guessing.