ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync skill updates and references

Browse Source
This commit is contained in:
root
2026-08-08 13:08:35 -04:00
parent b92542c8ae
commit 2f9b99b03a
33 changed files with 315 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/disaster-recovery-audit/SKILL.md
+1 -1
View File
@@ -214,7 +214,7 @@ For **backup pipeline failure patterns and integrity verification** — the Jul
```
┌──────────────────────┐ ┌───────────────────────┐
│ Core (netcup) │ │ app1-bu (Hetzner) │
│ 152.53.192.33 │ │ 5.161.114.8 │
│ 152.53.192.33 │ │ 5.161.225.131 │
│ RS 2000 (8C/16G) │ │ CPX11 (2C/2G) │
│ Active Hermes │ │ Dormant Hermes │
│ Active gateway │ │ Active watchdog │
skills/devops/disaster-recovery-audit/references/dr-audit-findings-2026-07-11.md
+2 -2
View File
@@ -23,13 +23,13 @@ fi
## Standby Server Disk Crisis
**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.
**Finding:** app1-bu (5.161.225.131) disk at 97% — only 1.2 GB free on 38 GB root partition.
**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.
**Required cleanup (not yet performed):**
```bash
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "
apt-get clean && apt-get autoremove --purge -y
journalctl --vacuum-size=200M
> /var/log/hermes-standby-sync.log
skills/devops/disaster-recovery-audit/references/itpp-recovery-manual.md
+10 -10
View File
@@ -12,7 +12,7 @@
1. [Architecture Overview](#1-architecture-overview)
2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233)
3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant)
4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148)
4. [app1-bu Standby (Hetzner — 5.161.225.131)](#4-app1-bu-standby-hetzner--51611148)
5. [Docker Services on Core](#5-docker-services-on-core)
- [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm)
- [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager)
@@ -43,7 +43,7 @@
| # | Server | IP | Provider | Specs | OS | What It Runs |
|---|--------|----|----------|-------|----|-------------|
| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN |
| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **2** | **app1-bu** (Standby) | 5.161.225.131 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant |
| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB |
| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB |
@@ -266,7 +266,7 @@ systemctl start hermes
---
## 4. app1-bu Standby (Hetzner — 5.161.114.8)
## 4. app1-bu Standby (Hetzner — 5.161.225.131)
**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.
@@ -274,7 +274,7 @@ systemctl start hermes
| Detail | Value |
|--------|-------|
| **IP** | 5.161.114.8 |
| **IP** | 5.161.225.131 |
| **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 (2C/2G/40G) |
| **OS** | Ubuntu 26.04 |
@@ -283,7 +283,7 @@ systemctl start hermes
### SSH Access
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra
```
### What Runs on Standby (Normally Dormant)
@@ -312,7 +312,7 @@ The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/sc
If you need to manually switch to standby:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
### How to Demote (Switch Back to Core)
@@ -320,7 +320,7 @@ ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
After Core is restored:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
```
Then make sure Hermes is running on Core:
@@ -1182,12 +1182,12 @@ bash /root/.hermes/scripts/run-wisp-backup.sh
2. **If automatic failover doesn't trigger**, do it manually:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
3. **Verify Hermes is running on standby:**
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
```
4. **Order a new netcup server:**
@@ -1320,7 +1320,7 @@ TZ=America/New_York
**Hermes internal cron jobs** (managed by Hermes itself):
- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json`
### On app1-bu (5.161.114.8)
### On app1-bu (5.161.225.131)
```
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
skills/devops/disaster-recovery-audit/references/network-and-service-endpoints.md
+1 -1
View File
@@ -26,7 +26,7 @@
| Name | Type | IP | Purpose |
|------|------|----|---------|
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby |
| app1-bu | CPX11 | 5.161.225.131 | Hermes warm standby |
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
| unms | CPX21 | 5.161.225.131 | UISP/UNMS |
| unifi | CPX21 | 178.156.131.57 | UniFi controller |
skills/devops/disaster-recovery-audit/references/server-dr-plans.md
+1 -1
View File
@@ -23,7 +23,7 @@
| 8 | docker | 178.156.168.35 | Hetzner | CPX11 | 🟢 LIVE | Utility Docker |
| 9 | n8n | 87.99.144.163 | Hetzner | CPX11 | 🟢 LIVE | Automation |
| 10 | tony-vps | 87.99.159.142 | Hetzner | CPX21 | 🟢 LIVE | Tony's Hermes |
| 11 | app1-bu | 5.161.114.8 | Hetzner | CPX11 | 🟢 STANDBY | Warm standby |
| 11 | app1-bu | 5.161.225.131 | Hetzner | CPX11 | 🟢 STANDBY | Warm standby |
## Full DR plans
See the full document at /root/.hermes/references/server-dr-plans.md for:
skills/devops/hermes-standby-deployment/references/hermes-dr-plan-v2.md
+1 -1
View File
@@ -9,7 +9,7 @@
## Architecture
```
Core (netcup RS 2000 — 152.53.192.33) app1-bu (Hetzner CPX21 — 5.161.114.8)
Core (netcup RS 2000 — 152.53.192.33) app1-bu (Hetzner CPX21 — 5.161.225.131)
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ Hermes Agent (active) │ │ Hermes Gateway (dormant) │
│ Ollama + llama3.2:3b (fallback) │◄─Tailnet─►│ Ollama + llama3.2:3b (standby) │
skills/devops/hermes-standby-deployment/references/standby-audit-2026-07-08.md
+1 -1
View File
@@ -1,6 +1,6 @@
# Standby Audit — July 8, 2026
## Server: app1-bu.itpropartner.com (5.161.114.8)
## Server: app1-bu.itpropartner.com (5.161.225.131)
Hetzner CPX11 (2 vCPU / 2 GB RAM / 40 GB SSD) — warm standby for app1 (152.53.192.33, netcup).
skills/devops/infrastructure-automation/references/destructive-action-protocol.md
+1 -1
View File
@@ -48,7 +48,7 @@ The complete policy stored in Hermes memory covers:
## Incidents that informed this policy
### Wrong-server reboot (Jul 5, 2026)
Accidentally rebooted `app1.itpropartner.com` (87.99.144.163, N8N+Postgres host, Hetzner ID 127781917) thinking it was `app1-bu` (5.161.114.8, dormant standby, Hetzner ID 125997675). The IDs are close enough to confuse in memory but one runs production databases. Root cause: relied on mental memory of opaque Hetzner API IDs instead of resolving them to hostname+IP first.
Accidentally rebooted `app1.itpropartner.com` (87.99.144.163, N8N+Postgres host, Hetzner ID 127781917) thinking it was `app1-bu` (5.161.225.131, dormant standby, Hetzner ID 125997675). The IDs are close enough to confuse in memory but one runs production databases. Root cause: relied on mental memory of opaque Hetzner API IDs instead of resolving them to hostname+IP first.
### Config key invention (prior to Jul 5)
Claimed `fallback_providers` was a real Hermes config option. It is not — Hermes silently accepts unknown YAML keys without validation. The key does nothing at runtime. User corrected with "don't be making shit up."
skills/devops/infrastructure-automation/references/disaster-recovery-plan.md
+1 -1
View File
@@ -54,7 +54,7 @@ app1 (netcup, live) app1-bu (Hetzner, standby)
3. Or reboot app1-bu — boot restore script runs S3 sync and starts Hermes
### Emergency failover (manual) — 2 min
1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.114.8`
1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.225.131`
2. Sync latest state: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock" --exclude "node/*" --exclude "bin/*" --exclude "cache/*" --exclude "sandboxes/*"`
3. Start Hermes: `hermes gateway start`
skills/devops/infrastructure-automation/references/hetzner-fleet-inventory.md
+1 -1
View File
@@ -6,7 +6,7 @@ Full server-by-server audit conducted 2026-07-09.
| Server | Hostname | IP | Plan | Specs | OS | Docker? | Key Apps |
|--------|----------|----|------|-------|----|---------|----------|
| app1-bu | app1-bu.itpropartner.com | 5.161.114.8 | CPX11 | 2C/2G/40G | Ubuntu 26.04 | ❌ | Standby Hermes, StrongSwan VPN |
| app1-bu | app1-bu.itpropartner.com | 5.161.225.131 | CPX11 | 2C/2G/40G | Ubuntu 26.04 | ❌ | Standby Hermes, StrongSwan VPN |
| ai.itpropartner.com | ai | 178.156.167.181 | CPX41 | 8C/16G/240G | Ubuntu 24.04 | ✅ 6 | LiteLLM, Open WebUI, Ollama, Qdrant |
| hudu.itpropartner.com | hudu | 178.156.130.130 | CPX21 | 3C/4G/80G | Ubuntu 24.04 | ✅ 5 | Hudu Docs, SWAG |
| unifi | unifi | 178.156.131.57 | CPX21 | 3C/4G/80G | Ubuntu 24.04 | ❌ | UniFi Controller, MongoDB |
skills/devops/infrastructure-automation/references/hetzner-server-audit-2026-07-12.md
+1 -1
View File
@@ -14,7 +14,7 @@ Complete audit of all Hetzner servers following the Core rebalance plan. Identif
| **docker** | 178.156.168.35 | CPX11 | ~$10 | Old Docker host | **Cancel now** — services migrated to netcup |
| **old-ai** | 178.156.167.181 | CPX41 | ~$30 | LitellM on 226G disk | **Cancel after LiteLLM migrates to Core** |
| **old app1** | 87.99.144.163 | CPX11 | ~$10 | Replaced by netcup app1 | **Cancel now** |
| **app1-bu** | 5.161.114.8 | CPX11 | ~$10 | Warm standby → core-bu | **Upgrade to CPX31 (4C/16G/160G, ~$22/mo)** for proper Core standby |
| **app1-bu** | 5.161.225.131 | CPX11 | ~$10 | Warm standby → core-bu | **Upgrade to CPX31 (4C/16G/160G, ~$22/mo)** for proper Core standby |
| **tony-vps** | 87.99.159.142 | CPX21 | ~$18 | Tony's personal Hermes | **Keep** — not our infrastructure to cancel |
## DNS Status
skills/devops/infrastructure-automation/references/l2tp-ipsec-debugging.md
+3 -3
View File
@@ -8,7 +8,7 @@ Real-world troubleshooting log from setting up a L2TP/IPsec VPN tunnel between a
```
parsed ID_PROT response 0 [ SA V V V V ]
no shared key found for '5.161.114.8' - '76.195.7.60'
no shared key found for '5.161.225.131' - '76.195.7.60'
received NO_PROPOSAL_CHOSEN error notify
```
@@ -18,7 +18,7 @@ received NO_PROPOSAL_CHOSEN error notify
**Fix:** Use explicit IPs in ipsec.secrets and strip YAML inline comments from the parsed value:
```bash
printf '5.161.114.8 %s : PSK "%s"\n' "$SERVER_IP" "$PSK" > "$IPSEC_SECRETS"
printf '5.161.225.131 %s : PSK "%s"\n' "$SERVER_IP" "$PSK" > "$IPSEC_SECRETS"
```
Also ensure the comment was stripped from the YAML value: `sed 's/ *#.*//'` after extraction.
@@ -26,7 +26,7 @@ Also ensure the comment was stripped from the YAML value: `sed 's/ *#.*//'` afte
### Symptom 2: "NO_PROPOSAL_CHOSEN" on ESP (Phase 2) after IKE SA established
```
IKE_SA wisp-vpn[5] established between 5.161.114.8[...]...76.195.7.60[...]
IKE_SA wisp-vpn[5] established between 5.161.225.131[...]...76.195.7.60[...]
parsed INFORMATIONAL_V1 request [ HASH N(NO_PROP) ]
received NO_PROPOSAL_CHOSEN error notify
```
skills/devops/infrastructure-automation/references/netcup-rs-g12-consolidation.md
+1 -1
View File
@@ -8,7 +8,7 @@ Replace 9 Hetzner Cloud VPSes (~$150/mo) with 2-4 Netcup RS G12 dedicated server
| Server | Type | Role | IP |
|--------|------|------|-----|
| `agent` | CPX11 | Hermes, VPN, backups | 5.161.114.8 |
| `agent` | CPX11 | Hermes, VPN, backups | 5.161.225.131 |
| `wphost02` | CPX21 | WordPress (RunCloud) | 5.161.62.38 |
| `unms.forefrontwireless.com` | CPX21 | UNMS (UISP) | 5.161.225.131 |
| `unifi` | CPX21 | UniFi Controller | 178.156.131.57 |
skills/devops/infrastructure-automation/references/netcup-server-provisioning.md
+1 -1
View File
@@ -141,7 +141,7 @@ rm -rf /tmp/aws /tmp/awscliv2.zip
### Wasabi S3 Credentials
```bash
aws configure set aws_access_key_id GYH83FP0KL0K85N60JKQ
aws configure set aws_access_key_id JGDE34XQVXTJKGAZIJYS
aws configure set aws_secret_access_key <key-from-control-box>
aws configure set region us-east-1
chmod 600 /root/.aws/credentials
skills/devops/infrastructure-automation/references/wisp-ccr-backup-pipeline.md
+1 -1
View File
@@ -149,7 +149,7 @@ python3 ~/.hermes/scripts/wisp-backup/wisp-backup.py
| Symptom | Likely fix |
|---------|-----------|
| VPN won't connect — IPsec negotiation reaches KE exchange then `no shared key found` | On strongSwan 6, the generic `: PSK "..."` format can fail. Use explicit IPs: `5.161.114.8 76.195.7.60 : PSK "..."`. Also ensure the secrets file doesn't have trailing YAML comments (use the sed strip-comment pattern). |
| VPN won't connect — IPsec negotiation reaches KE exchange then `no shared key found` | On strongSwan 6, the generic `: PSK "..."` format can fail. Use explicit IPs: `5.161.225.131 76.195.7.60 : PSK "..."`. Also ensure the secrets file doesn't have trailing YAML comments (use the sed strip-comment pattern). |
| `NO_PROPOSAL_CHOSEN` after IKE_SA established | ESP proposal mismatch. Add `-modp1024` to esp line: `esp=aes128-sha1-modp1024`. MikroTik L2TP/IPsec expects MODP group in ESP proposals. |
| pppd exits with `unrecognized option 'lock'` | pppd 2.5.x removed the `lock` option. Omit it from ppp options files. |
| VPN connects but internet dies, can't reach VPS | You used `replacedefaultroute` in the ppp options. Remove it — use `defaultroute` + explicit static routes instead. If it's already happening, disable the ppp secret on the MikroTik to restore connectivity, then fix the options file. |
skills/devops/mikrotik-onboarding/SKILL.md
+1 -1
View File
@@ -280,7 +280,7 @@ systemctl restart wg-quick@wg0
**Symptom:** `wg show wg0` shows TX increasing but RX stays at 0. The server sends keepalives (TX goes up) but the router never responds.
### Jul 9 post-migration: Three WG mismatches
When the server migrated from Hetzner (5.161.114.8) to netcup (152.53.192.33), the WG tunnel broke due to THREE simultaneous mismatches:
When the server migrated from Hetzner (5.161.225.131) to netcup (152.53.192.33), the WG tunnel broke due to THREE simultaneous mismatches:
1. **Stale router public key** — Server's `wg0.conf` used old key from `/root/wg-ccr.pub`, but the router had regenerated its WG private key since
2. **Stale server public key** — Router's peer endpoint still had the Hetzner box's public key, not the new Core's key
skills/devops/mikrotik-onboarding/references/home-router-reference.md
+1 -1
View File
@@ -31,7 +31,7 @@ The router has **two WireGuard peers** — one for Core (active Hermes) and one
### Peer 2 — app1-bu (Hetzner standby)
- **Peer:** 10.77.0.3 (app1-bu)
- **app1-bu public key:** `aEDfY/XzE5m6van/NkvMnYZoYuhptvs1GGkbqmaFKBQ=`
- **Endpoint:** 5.161.114.8:51821
- **Endpoint:** 5.161.225.131:51821
- **Keepalive:** 25s
**Latency:** 37ms (Ashburn → Savannah) for Core, 87ms for app1-bu (Hetzner → Savannah)
skills/devops/ops-portal-and-collector/SKILL.md
+1 -1
View File
@@ -707,7 +707,7 @@ fetch('/nav.html')
### Wasabi S3
- Endpoint: `https://s3.us-east-1.wasabisys.com`
- Credentials from `~/.aws/credentials` (key: `GYH83FP0KL0K85N60JKQ`)
- Credentials from `~/.aws/credentials` (key: `JGDE34XQVXTJKGAZIJYS`)
- CLI via `/opt/awscli-venv/bin/activate` then `aws s3 ls --recursive --endpoint-url ...`
- 3 buckets: `hermes-vps-backups`, `mikrotik-ccr-backups`, `itpropartner-backups`
skills/devops/ops-portal-and-collector/references/bulk-node-exporter-install.md
+1 -1
View File
@@ -6,7 +6,7 @@ All 9 Hetzner servers received node_exporter v1.8.2 in a single session:
| Hostname | IP | Status |
|----------|-----|--------|
| app1-bu.itpropartner.com | 5.161.114.8 | Running |
| app1-bu.itpropartner.com | 5.161.225.131 | Running |
| ai.itpropartner.com | 178.156.167.181 | Running |
| hudu.itpropartner.com | 178.156.130.130 | Running |
| (unifi) | 178.156.131.57 | Running |
skills/devops/ops-portal-and-collector/references/multi-server-health-api.md
+1 -1
View File
@@ -10,7 +10,7 @@ SERVERS = [
{"name": "app1", "ip": "152.53.36.131", "role": "AI Stack", "tier": "Standard"},
{"name": "app2", "ip": "152.53.39.202", "role": "UNMS, UniFi, Traccar", "tier": "Standard"},
{"name": "app3", "ip": "152.53.241.111", "role": "WordPress", "tier": "Standard"},
{"name": "app1-bu", "ip": "5.161.114.8", "role": "Warm Standby", "tier": "Standby"},
{"name": "app1-bu", "ip": "5.161.225.131", "role": "Warm Standby", "tier": "Standby"},
{"name": "AI Box (legacy)", "ip": "178.156.167.181", "role": "LiteLLM (migrating)", "tier": "Legacy"},
{"name": "Docker Box (legacy)", "ip": "178.156.168.35", "role": "Uptime Kuma", "tier": "Legacy"},
]
skills/devops/ops-portal-and-collector/templates/prometheus.yml
+1 -1
View File
@@ -41,6 +41,6 @@ scrape_configs:
- '178.156.149.32:9100' # fleettracker360
- '178.156.168.35:9100' # docker
- '178.156.167.181:9100' # ai
- '5.161.114.8:9100' # app1-bu
- '5.161.225.131:9100' # app1-bu
- '87.99.144.163:9100' # app1 (n8n)
- '87.99.159.142:9100' # tony-vps
skills/devops/reboot-with-health-check/SKILL.md
+1 -1
View File
@@ -89,7 +89,7 @@ This applies to ALL tools: terminal, write_file (config files), patch (productio
## Pitfalls
- **WRONG SERVER DISASTER.** Accidentally rebooted `app1.itpropartner.com` (N8N host, ID 127781917, IP 87.99.144.163) thinking it was `app1-bu` (standby, ID 125997675, IP 5.161.114.8). The IDs are close enough to confuse in memory (127781917 vs 125997675) but one runs N8N+Postgres and the other is a dormant standby. **What should have happened:** never reference servers by Hetzner API ID alone. Resolve the ID to hostname + IP, then state BOTH to the user before any power-cycle. Verifiable precondition: `python3 -c "import urllib.request,json; s=json.loads(urllib.request.urlopen(urllib.request.Request('https://api.hetzner.cloud/v1/servers/<ID>', headers={'Authorization':'Bearer ...'})).read())['server']; print(s['name'], s['public_net']['ipv4']['ip'])"` before any destructive action. Applies to app1-bu (standby), app1.itpropartner.com (N8N+Postgres), ai.itpropartner.com (LiteLLM proxy), and any future Hetzner box.
- **WRONG SERVER DISASTER.** Accidentally rebooted `app1.itpropartner.com` (N8N host, ID 127781917, IP 87.99.144.163) thinking it was `app1-bu` (standby, ID 125997675, IP 5.161.225.131). The IDs are close enough to confuse in memory (127781917 vs 125997675) but one runs N8N+Postgres and the other is a dormant standby. **What should have happened:** never reference servers by Hetzner API ID alone. Resolve the ID to hostname + IP, then state BOTH to the user before any power-cycle. Verifiable precondition: `python3 -c "import urllib.request,json; s=json.loads(urllib.request.urlopen(urllib.request.Request('https://api.hetzner.cloud/v1/servers/<ID>', headers={'Authorization':'Bearer ...'})).read())['server']; print(s['name'], s['public_net']['ipv4']['ip'])"` before any destructive action. Applies to app1-bu (standby), app1.itpropartner.com (N8N+Postgres), ai.itpropartner.com (LiteLLM proxy), and any future Hetzner box.
- **Always verify the server is NOT booted into rescue mode** (hostname check)
- **Don't assume SSH = all services** — specify critical service ports in the check
- **For rescue-mode reboots**, disable rescue FIRST, THEN reboot
skills/devops/server-architecture-plan/SKILL.md
+2 -2
View File
@@ -19,7 +19,7 @@ All new netcup servers are **RS 4000 G12** — 12 dedicated EPYC 9645 cores, 32
| **app1** | RS 4000 | ~$44/mo | 152.53.36.131 | **Service hub** — Caddy (all domains), Ops Portal, OSINT API, Super Search MCP, SearXNG, Grafana, cron, n8n + Postgres, Docker services |
| **app2** | RS 4000 | ~$44/mo | 152.53.39.202 | **Infrastructure** — UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target) |
| **app3** | RS 4000 | ~$44/mo | 152.53.241.111 | **Web apps** — WordPress/Apex from wphost02/SiteGround |
| **core-bu** (was app1-bu) | CPX11→CPX31 | ~$22/mo | 5.161.114.8 | Warm standby, needs upgrade for DR capacity |
| **core-bu** (was app1-bu) | CPX11→CPX31 | ~$22/mo | 5.161.225.131 | Warm standby, needs upgrade for DR capacity |
| | | **~$174/mo** | | |
### Hetzner Cleanup Plan (Jul 15, 2026 — mostly complete)
@@ -33,7 +33,7 @@ All new netcup servers are **RS 4000 G12** — 12 dedicated EPYC 9645 cores, 32
- fleettracker (178.156.149.32) — ✅ Deleted Jul 14
**Remaining Hetzner servers (4):**
- core-bu (5.161.114.8) — Warm standby, keep
- core-bu (5.161.225.131) — Warm standby, keep
- ai.itpropartner.com (178.156.167.181) — LiteLLM, pending migration to app1
- wphost02 (5.161.62.38) — WordPress, pending migration to app3
- app1.itpropartner.com (87.99.144.163) — Old app1, deprecated