chore: sync skill updates and references

2026-08-08 13:08:35 -04:00
parent b92542c8ae
commit 2f9b99b03a
33 changed files with 315 additions and 42 deletions
@@ -48,7 +48,7 @@ The complete policy stored in Hermes memory covers:
 ## Incidents that informed this policy

 ### Wrong-server reboot (Jul 5, 2026)
-Accidentally rebooted `app1.itpropartner.com` (87.99.144.163, N8N+Postgres host, Hetzner ID 127781917) thinking it was `app1-bu` (5.161.114.8, dormant standby, Hetzner ID 125997675). The IDs are close enough to confuse in memory but one runs production databases. Root cause: relied on mental memory of opaque Hetzner API IDs instead of resolving them to hostname+IP first.
+Accidentally rebooted `app1.itpropartner.com` (87.99.144.163, N8N+Postgres host, Hetzner ID 127781917) thinking it was `app1-bu` (5.161.225.131, dormant standby, Hetzner ID 125997675). The IDs are close enough to confuse in memory but one runs production databases. Root cause: relied on mental memory of opaque Hetzner API IDs instead of resolving them to hostname+IP first.

 ### Config key invention (prior to Jul 5)
 Claimed `fallback_providers` was a real Hermes config option. It is not — Hermes silently accepts unknown YAML keys without validation. The key does nothing at runtime. User corrected with "don't be making shit up."
@@ -54,7 +54,7 @@ app1 (netcup, live)              app1-bu (Hetzner, standby)
 3. Or reboot app1-bu — boot restore script runs S3 sync and starts Hermes

 ### Emergency failover (manual) — 2 min
-1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.114.8`
+1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.225.131`
 2. Sync latest state: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock" --exclude "node/*" --exclude "bin/*" --exclude "cache/*" --exclude "sandboxes/*"`
 3. Start Hermes: `hermes gateway start`

@@ -6,7 +6,7 @@ Full server-by-server audit conducted 2026-07-09.

 | Server | Hostname | IP | Plan | Specs | OS | Docker? | Key Apps |
 |--------|----------|----|------|-------|----|---------|----------|
-| app1-bu | app1-bu.itpropartner.com | 5.161.114.8 | CPX11 | 2C/2G/40G | Ubuntu 26.04 | ❌ | Standby Hermes, StrongSwan VPN |
+| app1-bu | app1-bu.itpropartner.com | 5.161.225.131 | CPX11 | 2C/2G/40G | Ubuntu 26.04 | ❌ | Standby Hermes, StrongSwan VPN |
 | ai.itpropartner.com | ai | 178.156.167.181 | CPX41 | 8C/16G/240G | Ubuntu 24.04 | ✅ 6 | LiteLLM, Open WebUI, Ollama, Qdrant |
 | hudu.itpropartner.com | hudu | 178.156.130.130 | CPX21 | 3C/4G/80G | Ubuntu 24.04 | ✅ 5 | Hudu Docs, SWAG |
 | unifi | unifi | 178.156.131.57 | CPX21 | 3C/4G/80G | Ubuntu 24.04 | ❌ | UniFi Controller, MongoDB |
@@ -14,7 +14,7 @@ Complete audit of all Hetzner servers following the Core rebalance plan. Identif
 | **docker** | 178.156.168.35 | CPX11 | ~$10 | Old Docker host | **Cancel now** — services migrated to netcup |
 | **old-ai** | 178.156.167.181 | CPX41 | ~$30 | LitellM on 226G disk | **Cancel after LiteLLM migrates to Core** |
 | **old app1** | 87.99.144.163 | CPX11 | ~$10 | Replaced by netcup app1 | **Cancel now** |
-| **app1-bu** | 5.161.114.8 | CPX11 | ~$10 | Warm standby → core-bu | **Upgrade to CPX31 (4C/16G/160G, ~$22/mo)** for proper Core standby |
+| **app1-bu** | 5.161.225.131 | CPX11 | ~$10 | Warm standby → core-bu | **Upgrade to CPX31 (4C/16G/160G, ~$22/mo)** for proper Core standby |
 | **tony-vps** | 87.99.159.142 | CPX21 | ~$18 | Tony's personal Hermes | **Keep** — not our infrastructure to cancel |

 ## DNS Status
@@ -8,7 +8,7 @@ Real-world troubleshooting log from setting up a L2TP/IPsec VPN tunnel between a

 ```
 parsed ID_PROT response 0 [ SA V V V V ]
-no shared key found for '5.161.114.8' - '76.195.7.60'
+no shared key found for '5.161.225.131' - '76.195.7.60'
 received NO_PROPOSAL_CHOSEN error notify
 ```

@@ -18,7 +18,7 @@ received NO_PROPOSAL_CHOSEN error notify

 **Fix:** Use explicit IPs in ipsec.secrets and strip YAML inline comments from the parsed value:
 ```bash
-printf '5.161.114.8 %s : PSK "%s"\n' "$SERVER_IP" "$PSK" > "$IPSEC_SECRETS"
+printf '5.161.225.131 %s : PSK "%s"\n' "$SERVER_IP" "$PSK" > "$IPSEC_SECRETS"
 ```

 Also ensure the comment was stripped from the YAML value: `sed 's/ *#.*//'` after extraction.
@@ -26,7 +26,7 @@ Also ensure the comment was stripped from the YAML value: `sed 's/ *#.*//'` afte
 ### Symptom 2: "NO_PROPOSAL_CHOSEN" on ESP (Phase 2) after IKE SA established

 ```
-IKE_SA wisp-vpn[5] established between 5.161.114.8[...]...76.195.7.60[...]
+IKE_SA wisp-vpn[5] established between 5.161.225.131[...]...76.195.7.60[...]
 parsed INFORMATIONAL_V1 request [ HASH N(NO_PROP) ]
 received NO_PROPOSAL_CHOSEN error notify
 ```
@@ -8,7 +8,7 @@ Replace 9 Hetzner Cloud VPSes (~$150/mo) with 2-4 Netcup RS G12 dedicated server

 | Server | Type | Role | IP |
 |--------|------|------|-----|
-| `agent` | CPX11 | Hermes, VPN, backups | 5.161.114.8 |
+| `agent` | CPX11 | Hermes, VPN, backups | 5.161.225.131 |
 | `wphost02` | CPX21 | WordPress (RunCloud) | 5.161.62.38 |
 | `unms.forefrontwireless.com` | CPX21 | UNMS (UISP) | 5.161.225.131 |
 | `unifi` | CPX21 | UniFi Controller | 178.156.131.57 |
@@ -141,7 +141,7 @@ rm -rf /tmp/aws /tmp/awscliv2.zip
 ### Wasabi S3 Credentials

 ```bash
-aws configure set aws_access_key_id GYH83FP0KL0K85N60JKQ
+aws configure set aws_access_key_id JGDE34XQVXTJKGAZIJYS
 aws configure set aws_secret_access_key <key-from-control-box>
 aws configure set region us-east-1
 chmod 600 /root/.aws/credentials
@@ -149,7 +149,7 @@ python3 ~/.hermes/scripts/wisp-backup/wisp-backup.py

 | Symptom | Likely fix |
 |---------|-----------|
-| VPN won't connect — IPsec negotiation reaches KE exchange then `no shared key found` | On strongSwan 6, the generic `: PSK "..."` format can fail. Use explicit IPs: `5.161.114.8 76.195.7.60 : PSK "..."`. Also ensure the secrets file doesn't have trailing YAML comments (use the sed strip-comment pattern). |
+| VPN won't connect — IPsec negotiation reaches KE exchange then `no shared key found` | On strongSwan 6, the generic `: PSK "..."` format can fail. Use explicit IPs: `5.161.225.131 76.195.7.60 : PSK "..."`. Also ensure the secrets file doesn't have trailing YAML comments (use the sed strip-comment pattern). |
 | `NO_PROPOSAL_CHOSEN` after IKE_SA established | ESP proposal mismatch. Add `-modp1024` to esp line: `esp=aes128-sha1-modp1024`. MikroTik L2TP/IPsec expects MODP group in ESP proposals. |
 | pppd exits with `unrecognized option 'lock'` | pppd 2.5.x removed the `lock` option. Omit it from ppp options files. |
 | VPN connects but internet dies, can't reach VPS | You used `replacedefaultroute` in the ppp options. Remove it — use `defaultroute` + explicit static routes instead. If it's already happening, disable the ppp secret on the MikroTik to restore connectivity, then fix the options file. |