Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,51 @@
|
||||
# Bulk SSH Key Injection via Hetzner Rescue Mode
|
||||
|
||||
When deploying the `itpp-infra` key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.
|
||||
|
||||
## Process
|
||||
|
||||
1. **Register the SSH key** in the Hetzner project (one-time per key, SKILL.md step 1)
|
||||
2. **Enable rescue mode + reboot on all targets** — batch via Python:
|
||||
```python
|
||||
targets = [(SERVER_ID, "hostname", "IP"), ...]
|
||||
for sid, name, ip in targets:
|
||||
data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
|
||||
req = urllib.request.Request(
|
||||
f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
|
||||
data=data.encode(), headers=headers, method='POST')
|
||||
urllib.request.urlopen(req, timeout=15)
|
||||
req2 = urllib.request.Request(
|
||||
f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
|
||||
data=b'{}', headers=headers, method='POST')
|
||||
urllib.request.urlopen(req2, timeout=15)
|
||||
```
|
||||
3. **Wait 60-90s** — poll SSH port on each:
|
||||
```bash
|
||||
for ip in "ip1" "ip2" ...; do
|
||||
timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
|
||||
done
|
||||
```
|
||||
4. **Inject key on all** — loop over each rescue host:
|
||||
```bash
|
||||
PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
|
||||
for ip in ...; do
|
||||
ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
|
||||
mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
|
||||
echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
|
||||
chmod 600 /mnt/root/.ssh/authorized_keys
|
||||
umount /mnt
|
||||
"
|
||||
done
|
||||
```
|
||||
5. **Disable rescue + reboot all** (same pattern as step 2, but `disable_rescue` + `reboot`)
|
||||
6. **Wait 90s**, then verify SSH access with `hostname` command
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Not all servers use `sda1`** — Hetzner rescue only has access to the raw block device. Check with `lsblk` on first SSH. On CPX11/CPX21 it's typically `sda1`; on systems with NVMe it could be `nvme0n1p1`. Fall back to `for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done`.
|
||||
- **Remount after first mount attempt** — If a partition is already auto-mounted from a previous rescue, the second `mount` fails. Check `mountpoint -q /mnt` before mounting.
|
||||
- **Boot order matters** — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
|
||||
- **`disable_rescue` BEFORE `reboot`** — Reboot with rescue still enabled just boots back into rescue.
|
||||
- **Server POST times vary** — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
|
||||
- **app1.itpropartner.com (87.99.144.163) has intermittent SSH delay** — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
|
||||
- **ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy** — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the `ssh_keys` parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue `root_password` from `enable_rescue` instead of key injection.
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# Failover Timing Update — July 8, 2026
|
||||
|
||||
## Change
|
||||
Per Germaine's direction, the failover timing was updated:
|
||||
|
||||
| Parameter | Before | After |
|
||||
|-----------|--------|-------|
|
||||
| Check interval | Every 10 min | Every 5 min |
|
||||
| Confirmation sleep | 60s × 4 cycles | 30s × 4 cycles |
|
||||
| Total confirmation window | ~3.5 min | ~2 min |
|
||||
| Max downtime before failover | ~13.5 min | ~7 min |
|
||||
|
||||
## What was changed
|
||||
1. **Cron on app1-bu:** `*/10 * * * *` → `*/5 * * * *`
|
||||
2. **Watchdog script sleep:** `sleep 60` → `sleep 30` (line ~71)
|
||||
3. **Confirmation message:** Updated from "~3.5 minutes" to "~2.0 minutes" (lines ~68, ~96)
|
||||
|
||||
## Rationale
|
||||
Faster detection = shorter failover. 7 min max vs 14 min max. User prioritized speed over cost.
|
||||
|
||||
## DR Plan impact
|
||||
DR-PLAN.md and server-inventory.md should be updated to reflect:
|
||||
- Server is warm (always on), not cold (boot on demand)
|
||||
- Check interval is 5 min, not 10 min
|
||||
- Verification needs to check both the cron schedule AND the sleep value
|
||||
@@ -0,0 +1,33 @@
|
||||
# Hermes DR Plan v2 — Pending Implementation Checklist
|
||||
|
||||
**Submitted by Germaine:** July 9, 2026
|
||||
**Status:** Planned (not yet implemented)
|
||||
**Supersedes:** v1 ping-only watchdog
|
||||
|
||||
## Key Architecture Changes
|
||||
|
||||
- Two-layer health check (ping + HTTP health endpoint)
|
||||
- SSH fencing + S3 ownership lock (prevents split-brain)
|
||||
- Pre-failover integrity check (SQLite PRAGMA)
|
||||
- 10-min VACUUM snapshots with integrity gate
|
||||
- Backward heartbeat (Core monitors standby)
|
||||
- app1-bu upgrade CPX11 → CPX21 for local Ollama (+$6/mo)
|
||||
|
||||
## 16-Item Checklist
|
||||
|
||||
- [ ] Upgrade app1-bu from CPX11 to CPX21 (Hetzner API)
|
||||
- [ ] Install Ollama + llama3.2:3b on app1-bu
|
||||
- [ ] Deploy /healthz endpoint on Core (Tailscale-only, bearer token)
|
||||
- [ ] Deploy two-layer watchdog on app1-bu (Layer 1 + Layer 2)
|
||||
- [ ] Deploy SSH fencing script on app1-bu
|
||||
- [ ] Deploy S3 ownership lock with refresh
|
||||
- [ ] Deploy 10-min VACUUM snapshot cron on Core
|
||||
- [ ] Deploy pre-failover integrity check on app1-bu
|
||||
- [ ] Deploy backward heartbeat (Core → app1-bu over Tailscale SSH)
|
||||
- [ ] Update hermes-standby-deployment skill with v2 architecture
|
||||
- [ ] Whitelist cross-box IPs in fail2ban both directions
|
||||
- [ ] Verify: 4 consecutive pings → auto-takeover (server dead)
|
||||
- [ ] Verify: Hermes dead but server up → alert + fence → conditional takeover
|
||||
- [ ] Verify: state.db corruption → falls back to snapshot, not corrupt file
|
||||
- [ ] Verify: Core reboot with standby holding lock → Core stays dormant
|
||||
- [ ] Update dr-issue-log.md
|
||||
@@ -0,0 +1,36 @@
|
||||
# Hermes Disaster Recovery Plan v2
|
||||
|
||||
**Last updated:** July 9, 2026
|
||||
**Author:** Sho'Nuff + Network Services Team
|
||||
**Supersedes:** `hermes-standby-deployment` skill's failover logic
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Core (netcup RS 2000 — 152.53.192.33) app1-bu (Hetzner CPX21 — 5.161.114.8)
|
||||
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
|
||||
│ Hermes Agent (active) │ │ Hermes Gateway (dormant) │
|
||||
│ Ollama + llama3.2:3b (fallback) │◄─Tailnet─►│ Ollama + llama3.2:3b (standby) │
|
||||
│ state.db (SQLite, ~1.87 GB) │ │ state.db (synced) │
|
||||
│ Telegram gateway (active) │ │ Telegram gateway (off) │
|
||||
│ │ │ │
|
||||
│ S3 backup every 15 min ─────────┼─────────►│ S3 sync every 5 min ◄───────────┤
|
||||
│ Snapshot every 10 min ──────────┼─────────►│ │
|
||||
└──────────────────────────────────┘ └──────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
s3://hermes-vps-backups/live/
|
||||
s3://hermes-vps-backups/snapshots/ (standby pulls snapshots on change)
|
||||
s3://hermes-vps-backups/ACTIVE_OWNER (standby writes on takeover only)
|
||||
```
|
||||
|
||||
**Key changes from v1:**
|
||||
- app1-bu upgraded from CPX11 (2C/2G) to CPX21 (4C/8G, ~$14/mo) to run its own local Ollama
|
||||
- Both boxes have local Ollama (llama3.2:3b) — health checks never depend on external LLM providers
|
||||
- Traffic between boxes goes over Tailscale tailnet (not public IPs), avoiding the Hetzner↔netcup routing issue
|
||||
- No 60-second S3 writes from standby — all cross-box communication uses Tailscale directly
|
||||
- S3 is backup storage only, not a communication channel
|
||||
|
||||
[Full document contents at /root/.hermes/cache/documents/doc_0e07de339e65_hermes-dr-plan-v2.md]
|
||||
@@ -0,0 +1,59 @@
|
||||
# Standby Audit — July 8, 2026
|
||||
|
||||
## Server: app1-bu.itpropartner.com (5.161.114.8)
|
||||
|
||||
Hetzner CPX11 (2 vCPU / 2 GB RAM / 40 GB SSD) — warm standby for app1 (152.53.192.33, netcup).
|
||||
|
||||
### Power State
|
||||
- Hetzner API: `running` (server is powered on, OS up)
|
||||
- Per DR plan: documented as "🟢 Dormant" but the intent in `hermes-standby-deployment` skill says "Stay powered on with Hermes off" — so running is **correct**, but server-inventory.md calls it "dormant" which is ambiguous.
|
||||
- **Note:** If electricity cost ($5/mo) is a concern, this could be a cold standby (boot on demand) with a ~60s slower failover.
|
||||
|
||||
### Hermes Gateway
|
||||
- `systemctl is-active hermes-gateway` → `inactive` ✅
|
||||
|
||||
### Standby Service
|
||||
- `systemctl is-active hermes-standby` → `active (exited)` ✅
|
||||
- Last boot: Jul 5, 2026
|
||||
- Uptime at audit: 3 days, 1:16
|
||||
- Restore log: `Live box is REACHABLE — staying dormant (clean exit)` ✅
|
||||
|
||||
### Watchdog Cron
|
||||
- `crontab -l` → `*/10 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh` ✅
|
||||
- Watchdog log: empty (no failures detected) ✅
|
||||
- Disk usage: 16G / 38G (43%)
|
||||
|
||||
### SSH Access
|
||||
- Key: `itpp-infra` (Ed25519) ✅
|
||||
- Recent auth log shows SSH from 152.53.192.33 (live box) — heartbeat or previous manual check
|
||||
|
||||
### S3 Artifacts
|
||||
- `s3://hermes-vps-backups/standby/` — all 6 files present ✅
|
||||
- `DR-PLAN.md` — 13 KB, present ✅
|
||||
- `hermes-standby-restore.sh` — 2.8 KB, matches deployed version ✅
|
||||
- `hermes-standby-watchdog.sh` — 4.3 KB, matches deployed version ✅
|
||||
- `hermes-standby.service` — 367 B, matches deployed version ✅
|
||||
- `server-inventory.md` — 8.1 KB, present ✅
|
||||
- `recovery-bundle-2026-07-05.md` — 280 KB, present ✅
|
||||
|
||||
### Live Sync Freshness
|
||||
- `hermes-live-sync` cron runs every 15 min on live box ✅
|
||||
- `state.db` on S3: 2026-07-08T23:19:08Z (~5 min from audit time) ✅
|
||||
- `gateway_state.json`: 2026-07-08T23:19:06Z ✅
|
||||
- `channel_directory.json`: 2026-07-08T23:18:59Z ✅
|
||||
- Total S3 `live/` data: 32,151 files / 4.8 GiB
|
||||
|
||||
### Credential Observations
|
||||
- `HETZNER_API_TOKEN` exists as a shell env var
|
||||
- `/root/.hermes/scripts/.hetzner_token` file is **missing** (deleted during prior cleanup?)
|
||||
- Recovery bundle references the file approach — potential doc/setup drift
|
||||
|
||||
### Issues Found
|
||||
1. Minor: server-inventory.md says "dormant" (implies off) but server is running — warm standby is correct per skill, but docs ambiguous
|
||||
2. Minor: `.hetzner_token` file deleted; token only exists in env var — recovery bundle instructs restoring the file
|
||||
3. No critical issues
|
||||
|
||||
### Next Audit Recommendation
|
||||
- Quarterly or after any DR-plan changes
|
||||
- Also verify the live-sync script hasn't drifted (exclusion patterns, credentials)
|
||||
- Check Hetzner snapshot cron (`hetzner-weekly-snapshots`) still runs
|
||||
@@ -0,0 +1,29 @@
|
||||
# Standby Hostname & OS Renaming
|
||||
|
||||
After deploying the standby watchdog script, also rename the standby's OS to match its DNS name.
|
||||
|
||||
## On the standby box
|
||||
|
||||
```bash
|
||||
hostnamectl set-hostname app1-bu.itpropartner.com
|
||||
```
|
||||
|
||||
Verify with both `hostname` and `hostname -f`.
|
||||
|
||||
## Also rename in the cloud provider API
|
||||
|
||||
The cloud provider labels (Hetzner Cloud Console, API `server.name`) and the OS hostname are separate. Update both:
|
||||
|
||||
```python
|
||||
import urllib.request, json
|
||||
headers = {'Authorization': f'Bearer {token}', 'Content-Type': 'application/json'}
|
||||
data = json.dumps({"name": "app1-bu.itpropartner.com"})
|
||||
req = urllib.request.Request(f'https://api.hetzner.cloud/v1/servers/{SERVER_ID}', data=data.encode(), headers=headers, method='PUT')
|
||||
urllib.request.urlopen(req, timeout=10)
|
||||
```
|
||||
|
||||
## Why this matters
|
||||
|
||||
- The DNS A record, the cloud provider name, and the OS hostname should all agree
|
||||
- Scripts that check `hostname` for routing decisions won't break
|
||||
- The DR documentation references the DNS name — matching OS hostname reduces confusion during failover
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
# Tailscale Serve for Internal Services
|
||||
|
||||
Standard way to expose Docker services on app1 to the tailnet without public exposure.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Install Tailscale
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
|
||||
# Authenticate — prints a URL to open in browser
|
||||
tailscale up
|
||||
```
|
||||
|
||||
Once authenticated, the server gets a `100.x.x.x` Tailscale IP:
|
||||
|
||||
```bash
|
||||
tailscale status
|
||||
```
|
||||
|
||||
## Expose a Service via Tailscale Serve
|
||||
|
||||
```bash
|
||||
# Expose port 8080 (Vaultwarden) as HTTPS via Tailscale
|
||||
tailscale serve --bg --https 443 --set-path / http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
This creates a URL like `https://<hostname>.tailc2f3b0.ts.net/` — only accessible within your tailnet, with a valid Tailscale TLS cert. No browser warnings.
|
||||
|
||||
## Rename Tailscale hostname
|
||||
|
||||
```bash
|
||||
tailscale set --hostname <name>
|
||||
```
|
||||
|
||||
The hostname becomes part of the Tailscale Serve URL.
|
||||
|
||||
## Switch URL without breaking the app
|
||||
|
||||
If DNS naming changes after Tailscale Serve is configured:
|
||||
|
||||
1. The service's DOMAIN env var in the Docker compose must match the Tailscale URL
|
||||
2. Update the compose file, restart the container
|
||||
3. The Bitwarden app on client devices needs the server URL updated to match
|
||||
|
||||
## UFW: Tailscale traffic only
|
||||
|
||||
```bash
|
||||
# Allow only tailscale interface for internal services
|
||||
ufw allow in on tailscale0 to any port 8080 proto tcp comment 'Service via Tailscale'
|
||||
|
||||
# Block public access
|
||||
ufw delete allow 8080/tcp
|
||||
ufw reload
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Internal services do NOT need public DNS or Let's Encrypt — Tailscale Serve provides HTTPS automatically
|
||||
- The Tailscale IP (`100.x.x.x`) changes if the node re-registers — always use the Tailscale domain name for stable references
|
||||
- Tailscale must be running and authenticated before Serve will work
|
||||
- On the standby/backup, Tailscale is optional but recommended for SSH access if public SSH is ever locked down
|
||||
|
||||
## Per-Device Setup
|
||||
|
||||
Install the Tailscale app on each device:
|
||||
- **iPhone/iPad:** App Store → "Tailscale"
|
||||
- **Mac:** [tailscale.com/download-mac](https://tailscale.com/download-mac)
|
||||
- Login with the same identity provider as the server
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **HTTPS vs HTTP on iOS Bitwarden app:** The Bitwarden app for iOS sometimes has trouble with Tailscale Serve HTTPS URLs. Workaround is to clear the app's server config, force-close, and re-add. If the problem persists, the app may cache encryption parameters tied to the old server URL — delete and reinstall the app.
|
||||
- **Hostname with zeros/Os:** If the Tailscale hostname contains `0` (zero) or `O` (letter O), the iOS Bitwarden app may confuse them when displaying the URL. Avoid this by choosing a hostname without ambiguous characters.
|
||||
- **Server URL change invalidates login:** If the DOMAIN env var changes in Vaultwarden, users will get "username or password is incorrect" errors on existing client sessions even with the correct password. The client must be reconfigured with the new Server URL. The existing session on the old URL does not carry over — this is a client-side cache issue, not an actual password mismatch.
|
||||
Reference in New Issue
Block a user