Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,110 @@
|
||||
# Apex WPForms Mail Debugging
|
||||
|
||||
Quick reference for diagnosing and fixing Apex Track Experience WPForms email delivery failures.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Value |
|
||||
|-----------|-------|
|
||||
| Server | wphost02 (5.161.62.38, Hetzner CPX21) |
|
||||
| DB | MySQL via RunCloud |
|
||||
| SMTP | c1113726.sgvps.net:2525, STARTTLS |
|
||||
| SMTP user | contact@apextrackexperience.com |
|
||||
| SMTP pass | apex.track!! |
|
||||
| Plugin | WP Mail SMTP (Lite) |
|
||||
| Forms | NASA registration (270), Waiver (268) |
|
||||
| Last known SMTP creds | Username: `contact@apextrackexperience.com`, Password: `apex.track!!` |
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
### 1. PHP Serialized Password Mismatch (most common cause)
|
||||
|
||||
WP Mail SMTP stores the SMTP password in the `wp_options` table as `wp_mail_smtp` in PHP serialized format. If the declared string length doesn't match the actual string, the plugin silently fails to authenticate.
|
||||
|
||||
**Symptoms:** Form submissions appear successful in the UI, but no email is sent. The debug events table shows `event_type = 0` with no useful error text.
|
||||
|
||||
**Check command:**
|
||||
```sql
|
||||
SELECT JSON_EXTRACT(option_value, '$.smtp.pass') FROM wp_options WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
The JSON_EXTRACT result shows the raw serialized PHP value. A broken entry looks like:
|
||||
```
|
||||
s:72:\"apex.track!!\"
|
||||
-- length 72 declared, but 'apex.track!!' is only 13 characters
|
||||
```
|
||||
|
||||
**Fix command:**
|
||||
```sql
|
||||
UPDATE wp_options SET option_value = REPLACE(option_value, 's:72:\"apex.track!!\"', 's:13:\"apex.track!!\"') WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
### 2. WPForms sender_address
|
||||
|
||||
Both forms had `contact@apextrackexperience.com, g@germainebrown.com` as the `sender_address`. This is invalid — `From:` headers require a single address. An SMTP server receiving two comma-separated addresses may reject the `MAIL FROM` command.
|
||||
|
||||
**Check command (form 270 and 268):**
|
||||
```sql
|
||||
SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications') AS notifs FROM wp_posts WHERE ID IN (270, 268);
|
||||
```
|
||||
|
||||
**Fix:** Set `sender_address` to `contact@apextrackexperience.com` only. The notification recipient `g@germainebrown.com` is configured separately as an email notification destination, not the sender.
|
||||
|
||||
### 3. SMTP Network Reachability
|
||||
|
||||
From wphost02:
|
||||
```bash
|
||||
# Test connection (not sending mail)
|
||||
timeout 5 bash -c 'echo | openssl s_client -connect c1113726.sgvps.net:2525 -starttls smtp'
|
||||
```
|
||||
|
||||
The `CONNECTED` line should appear in the output.
|
||||
|
||||
### 4. Debug Events Table
|
||||
|
||||
WP Mail SMTP logs email delivery attempts to `wp_wpmailsmtp_debug_events`:
|
||||
```sql
|
||||
SELECT id, subject, status, date_sent, error_text FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 5;
|
||||
```
|
||||
|
||||
- `event_type = 0` — email sent (or attempted)
|
||||
- `event_type = 1` — email failed
|
||||
- Check `created_at >= NOW() - INTERVAL 10 MINUTE` for recent attempts
|
||||
|
||||
## Watchdog
|
||||
|
||||
A cron job on Core monitors Apex mail every 5 minutes:
|
||||
- `/root/.hermes/scripts/apex-mail-watchdog.sh`
|
||||
- Tests SMTP LOGIN (does NOT send a test email — only verifies credentials)
|
||||
- Checks debug events for recent failures
|
||||
- Silent on success, alerts Germaine on failure
|
||||
- Do NOT use `tee -a` in the log function — it causes every log line to be delivered as a cron message
|
||||
|
||||
## Watchdog Blindness (SSH-dependency failure mode)
|
||||
|
||||
The watchdog runs FROM Core and SSHes to wphost02 (`root@5.161.62.38`, key `/root/.ssh/itpp-infra`, port 22) to test SMTP and read the debug table. Both steps require that SSH hop. **If SSH is refused/unreachable, the watchdog keeps "running" every 5 min but tests NOTHING** — the log fills with:
|
||||
|
||||
```
|
||||
SMTP FAILED: ssh: connect to host 5.161.62.38 port 22: Connection refused
|
||||
```
|
||||
|
||||
This is a monitoring blind spot, not a mail failure: the watchdog cannot distinguish "SMTP broken" from "I can't reach the box." When auditing Apex mail, ALWAYS confirm the SSH hop works before trusting a "FAIL" verdict:
|
||||
|
||||
```bash
|
||||
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=5 root@5.161.62.38 'echo reachable'
|
||||
tail -5 /var/log/apex-mail-watchdog.log # look for "connect to host ... Connection refused"
|
||||
```
|
||||
|
||||
If SSH is refused: the box may be down, rebooting, mid-migration, or its SSH port/IP changed. **During the Hetzner->netcup migration this is expected churn** — wphost02 (Apex/WordPress) is slated to move to app3, which changes IP and possibly port. When a WordPress/mail host migrates, update `WPHOST` and any port in `apex-mail-watchdog.sh` and re-verify the hop. Do not report "Apex mail is down" from a `Connection refused` line alone.
|
||||
|
||||
## Watching the Watchdog
|
||||
|
||||
```bash
|
||||
# Check the watchdog cron
|
||||
cronjob action=list | grep apex
|
||||
|
||||
# View the watchdog log
|
||||
cat /var/log/apex-mail-watchdog.log
|
||||
|
||||
# The ops portal at ops.itpropartner.com/cron.html shows the job status live
|
||||
```
|
||||
@@ -0,0 +1,69 @@
|
||||
# Apex WPForms Mail Fix (Post-Outage)
|
||||
|
||||
When Apex Track Experience form notifications (register/waiver) stop sending, follow this checklist:
|
||||
|
||||
## 1. Check Debug Events
|
||||
|
||||
```sql
|
||||
SELECT id, subject, status, date_sent, error_text
|
||||
FROM wp_wpmailsmtp_debug_events
|
||||
ORDER BY id DESC LIMIT 10;
|
||||
```
|
||||
|
||||
## 2. Check SMTP Password Serialization
|
||||
|
||||
The WP Mail SMTP plugin stores passwords in PHP serialized format in wp_options:
|
||||
|
||||
```sql
|
||||
SELECT option_value FROM wp_options WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
Look for: `s:72:"apex.track!!"` - the length prefix (72) must match actual password length (13).
|
||||
|
||||
**Fix:**
|
||||
```sql
|
||||
UPDATE wp_options
|
||||
SET option_value = REPLACE(option_value,
|
||||
's:72:\\"apex.track!!\\"',
|
||||
's:13:\\"apex.track!!\\"')
|
||||
WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
## 3. Check Form Sender Address
|
||||
|
||||
WPForms notification sender_address should be a single email, not comma-separated.
|
||||
|
||||
```sql
|
||||
SELECT ID, JSON_EXTRACT(post_content, '$.settings.notifications."1".sender_address') as sender
|
||||
FROM wp_posts WHERE ID IN (270, 268) AND post_type = 'wpforms';
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```sql
|
||||
UPDATE wp_posts SET post_content = REPLACE(
|
||||
post_content,
|
||||
'contact@apextrackexperience.com, g@germainebrown.com',
|
||||
'contact@apextrackexperience.com'
|
||||
) WHERE ID IN (270, 268);
|
||||
```
|
||||
|
||||
## 4. Re-send Missed Notifications
|
||||
|
||||
```sql
|
||||
SELECT entry_id, form_id, fields FROM wp_wpforms_entries
|
||||
WHERE DATE(date) = CURDATE() AND form_id IN (268, 270, 272);
|
||||
```
|
||||
|
||||
Re-send via SMTP Python script using the Apex SMTP server at c1113726.sgvps.net:2525.
|
||||
|
||||
## 5. Verify SMTP Connectivity
|
||||
|
||||
Test from the server:
|
||||
```python
|
||||
import smtplib, ssl
|
||||
ctx = ssl.create_default_context()
|
||||
with smtplib.SMTP('c1113726.sgvps.net', 2525, timeout=15) as s:
|
||||
s.starttls(context=ctx)
|
||||
s.login('contact@apextrackexperience.com', 'apex.track!!')
|
||||
print('OK')
|
||||
```
|
||||
@@ -0,0 +1,41 @@
|
||||
## Apex WPForms SMTP Serialization Bug
|
||||
|
||||
**Root cause:** WP Mail SMTP stores passwords in PHP serialized format within the `wp_options` table. The serialized string has a length prefix (e.g., `s:13:` for 13 characters). If the declared length doesn't match the actual password string length, the plugin cannot deserialize the password and **ALL form email delivery fails silently**.
|
||||
|
||||
**Affected configuration:** `option_name = 'wp_mail_smtp'`, stored in `wp_options` table.
|
||||
|
||||
**The bug:** The password `apex.track!!` is 13 characters. The stored value had `s:72` instead of `s:13` — likely caused by a pre-save sanitization or paste issue. WP Mail SMTP read `s:72` and expected a 72-character string, found fewer characters, and failed with a PHP warning that never surfaced to the UI.
|
||||
|
||||
**Diagnosis:**
|
||||
1. Check the stored password: `SELECT option_value FROM wp_options WHERE option_name = 'wp_mail_smtp'`
|
||||
2. Search for `smtp.pass` in the serialized data
|
||||
3. Count the actual password length and compare to the serialized `s:N` prefix
|
||||
|
||||
**Fix (verified Jul 8, 2026):**
|
||||
```sql
|
||||
UPDATE wp_options
|
||||
SET option_value = REPLACE(
|
||||
option_value,
|
||||
's:72:"apex.track!!"',
|
||||
's:13:"apex.track!!"'
|
||||
)
|
||||
WHERE option_name = 'wp_mail_smtp';
|
||||
```
|
||||
|
||||
**Also fix sender_address on forms:**
|
||||
Both Form 270 (NASA Top Speed) and Form 268 (Waiver) had comma-separated email addresses in the `sender_address` notification field. This produces an invalid `From:` email header. Set `sender_address` to a single email address (e.g., `contact@apextrackexperience.com`).
|
||||
|
||||
**Verification:**
|
||||
1. Send a test email from the WP Mail SMTP settings page
|
||||
2. Check debug events table: `SELECT * FROM wp_wpmailsmtp_debug_events ORDER BY id DESC LIMIT 10`
|
||||
3. Submit a test WPForms entry and verify the notification arrives
|
||||
4. Enable the 5-min watchdog script
|
||||
|
||||
**Resending missed notifications:**
|
||||
After fixing the serialization, previously failed form entries won't auto-resend. Check `wp_wpforms_entries` for entries with no corresponding email notification. Manually re-send via SMTP Python script.
|
||||
|
||||
**Watchdog script** at `/root/.hermes/scripts/apex-mail-watchdog.sh` runs every 5 minutes via no_agent cron. It:
|
||||
- Tests SMTP LOGIN authenticity (no test email sent — just `s.login()` then `s.quit()`)
|
||||
- Queries `wp_wpmailsmtp_debug_events` for recent failures
|
||||
- Silent on success, alerts on failure
|
||||
- **CRITICAL:** `log()` function uses `>> "$LOG"` not `tee -a "$LOG"` to prevent every log line from being delivered as a cron message
|
||||
@@ -0,0 +1,67 @@
|
||||
# DR Audit Findings — Jul 11, 2026
|
||||
|
||||
## Backup Pipeline Failure
|
||||
|
||||
**Trigger:** Daily full backup (1 AM UTC) and root-essentials backup (3 AM UTC) both failed silently on Jul 11.
|
||||
|
||||
**Root cause:** `hermes-backup.sh` was missing `source /opt/awscli-venv/bin/activate`. The `aws` command is only available inside the virtualenv at `/opt/awscli-venv/`. In cron's minimal PATH (`/usr/bin:/bin`), `aws` is not found. The `root-essentials-backup.sh` script HAS the venv activation line but also failed — likely the same issue or a system-level path problem for tar appends.
|
||||
|
||||
**Evidence from journalctl:**
|
||||
```
|
||||
Jul 11 01:00:46 core hermes-full-backup[19447]: /root/.hermes/scripts/hermes-backup.sh: line 215: aws: command not found
|
||||
```
|
||||
|
||||
**Fix applied (Jul 11, 07:30 UTC):**
|
||||
```bash
|
||||
# Added after "set -euo pipefail" in hermes-backup.sh:
|
||||
if [ -f /opt/awscli-venv/bin/activate ]; then
|
||||
source /opt/awscli-venv/bin/activate
|
||||
fi
|
||||
```
|
||||
|
||||
**Verification:** Manual backup ran successfully after fix — 541 MB tarball uploaded to `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-11.tar.gz`.
|
||||
|
||||
## Standby Server Disk Crisis
|
||||
|
||||
**Finding:** app1-bu (5.161.114.8) disk at 97% — only 1.2 GB free on 38 GB root partition.
|
||||
|
||||
**Impact:** If Core had failed, the standby wouldn't have had disk space to complete a failover (state.db is ~1.9 GB, plus logs and temp files). The watchdog script would crash mid-sync.
|
||||
|
||||
**Required cleanup (not yet performed):**
|
||||
```bash
|
||||
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
|
||||
apt-get clean && apt-get autoremove --purge -y
|
||||
journalctl --vacuum-size=200M
|
||||
> /var/log/hermes-standby-sync.log
|
||||
docker system prune -a -f 2>/dev/null || true
|
||||
df -h /
|
||||
"
|
||||
```
|
||||
Target: at least 5 GB free.
|
||||
|
||||
## Backup Integrity Verification Workflow
|
||||
|
||||
The Jul 11 audit established a reliable pattern for verifying backup integrity:
|
||||
|
||||
1. **List S3** to find the latest tarball
|
||||
2. **Download** it to `/tmp/` with `aws s3 cp`
|
||||
3. **Test** with `tar tzf` (not just `aws s3 ls`)
|
||||
4. **Report** file count and any corruption
|
||||
5. **Clean up** the local copy
|
||||
|
||||
This catches silent failures that S3 listing alone misses. The audit watchdog (`backup-audit-check.sh`) only checks that a file EXISTS — a zero-byte or corrupted tarball passes that check.
|
||||
|
||||
## Restore Plan Template
|
||||
|
||||
When an incident requires a restore plan, structure it with these sections:
|
||||
|
||||
1. **Pre-Restore Assessment** — backup status table, standby status, system health, dependency inventory
|
||||
2. **Recovery Path A** — In-place fix (preferred when system is running)
|
||||
3. **Recovery Path B** — Failover to standby (when Core is unreachable)
|
||||
4. **Recovery Path C** — Full rebuild from backup tarball (when OS is corrupted)
|
||||
5. **Post-Restore Verification Checklist** — Hermes, web services, model access, email, backups, standby
|
||||
6. **Critical Fixes** — any permanent fixes needed to prevent recurrence
|
||||
7. **Team Contact & Escalation** — who to call and when
|
||||
8. **Rollback Plan** — how to undo the restore if it makes things worse
|
||||
|
||||
See `/root/.hermes/references/restore-plan-2026-07-11.md` for the complete worked example.
|
||||
@@ -0,0 +1,82 @@
|
||||
# DR Issue Log — Hermes Infrastructure
|
||||
|
||||
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-08 — Initial Full DR Audit
|
||||
|
||||
### DR-001: Caddyfile not included in backup scripts
|
||||
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
|
||||
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
|
||||
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates).
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed all paths in script
|
||||
|
||||
### DR-002: Systemd service files backed up to wrong path
|
||||
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up.
|
||||
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
|
||||
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed paths in script
|
||||
|
||||
### DR-003: migration-creds.txt not in backup scope
|
||||
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
|
||||
**Root cause:** Added after backup script was written, never included in scope.
|
||||
**Fix:** Added to hermes-backup.sh file list with chmod 600 restore.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** bash -n syntax check + subagent confirmed file referenced
|
||||
|
||||
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
|
||||
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
|
||||
**Root cause:** Script created the file without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-005: migration-creds.txt at 644 permissions
|
||||
**Problem:** Same issue as DR-004 — migration-creds.txt was 644.
|
||||
**Root cause:** Written without explicit permission setting.
|
||||
**Fix:** chmod 600
|
||||
**Status:** ✅ Fixed 2026-07-08
|
||||
**Verified by:** ls -la confirms 600
|
||||
|
||||
### DR-006: Full backup had no cron job (last ran Jul 5)
|
||||
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
|
||||
**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
|
||||
**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Cron job created and scheduled for next 5 AM ET
|
||||
|
||||
### DR-007: home-router-daily-backup cron error
|
||||
**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed.
|
||||
**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP.
|
||||
**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||||
|
||||
### DR-008: MikroTik CCR backup stale (last Jul 5)
|
||||
**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5.
|
||||
**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24.
|
||||
**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan).
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||||
|
||||
### DR-009: app1-bu warm vs cold docs mismatch
|
||||
**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime).
|
||||
**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant).
|
||||
**Fix:** Updated DR plan to reflect warm standby design. Server stays running.
|
||||
**Status:** ✅ Resolved — not a bug, docs were wrong
|
||||
|
||||
### DR-010: Failover timing change (per Germaine's direction)
|
||||
**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles).
|
||||
**Rationale:** Faster detection, shorter failover.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu.
|
||||
|
||||
### DR-011: AWS CLI missing on app1-bu standby
|
||||
**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3.
|
||||
**Root cause:** python3-venv package not installed on standby server.
|
||||
**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron.
|
||||
**Status:** ✅ Fixed & verified 2026-07-08
|
||||
**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.
|
||||
@@ -0,0 +1,43 @@
|
||||
# DR Plan v3 — Response to Third-Party Reviewers
|
||||
|
||||
## Two reviewers, two passes. All findings addressed.
|
||||
|
||||
### Reviewer 1 (Technical DR Specialist)
|
||||
|
||||
| Finding | Status | Addressed In |
|
||||
|---|---|---|
|
||||
| Failover timing mismatch (5min check can't detect 2min outage) | Fixed | Corrected to 30s checks, 4 failures to trigger = 2min detection, ~3-3.5min actual RTO, 5min committed target |
|
||||
| Backup standard says daily but not happening | Flagged | Section 3.1 backup matrix shows live vs gap status; P0 items for Docker volume + DB dump automation |
|
||||
| AI server 92% disk critical | Flagged | P0 action item #1 |
|
||||
| Memory consolidation data loss risk | Mitigated | S3 backup-before-prune + PROTECT regex expansion + restore procedure documented |
|
||||
| Tony's Hermes undocumented | Flagged | P1 action item #7 |
|
||||
|
||||
### Reviewer 2 (Systems Management SME)
|
||||
|
||||
| Finding | Status | Addressed In |
|
||||
|---|---|---|
|
||||
| S3 single point of failure | Mitigated | Two-destination design (Wasabi us-east-1 primary + Wasabi us-west-2 or Backblaze B2 secondary) |
|
||||
| Backup frequency vs RPO mismatch | Fixed | Per-data-type RPO: 15min (Hermes state), 24h (Docker/data), 24h backups |
|
||||
| Restore procedures not detailed | Fixed | Section 6 — per-server runbooks with exact commands |
|
||||
| Failback under-defined | Fixed | Section 4.4 — 12-step protocol with rollback, no auto-failback |
|
||||
| No backup restore testing | Fixed | Section 7 — testing schedule with cadence, log template, sign-off table |
|
||||
| Retention too short (48h snapshots) | Flagged | P1 item #9 — extend to 30 days |
|
||||
| Backup security not described | Fixed | Section 3.2 — Object Lock, write-only IAM, credential separation |
|
||||
| Application backup details vague | Partial | Section 2.3 — data paths defined per app |
|
||||
| Secrets recovery not documented | Fixed | Section 1 — shared dependencies table with credential locations |
|
||||
| Provider/account outage missing | Fixed | Section 4.5 — netcup account failure scenario |
|
||||
|
||||
## Still Pending (Not Yet Implemented, Only Designed)
|
||||
|
||||
| Item | Status | Required For Sign-Off |
|
||||
|---|---|---|
|
||||
| Object Lock enabled on buckets | Designed, not done | Yes |
|
||||
| Write-only IAM credentials created | Designed, not done | Yes |
|
||||
| S3 secondary backup destination provisioned | Designed, not done | Yes |
|
||||
| Docker volume backup automation deployed | Flagged, not done | Yes |
|
||||
| MariaDB dump cron deployed (wphost02, fleettracker360) | Flagged, not done | Yes |
|
||||
| 30-second watchdog deployed on app1-bu | Designed, not done | Yes |
|
||||
| Tony's Hermes documented + backed up | Flagged, not done | No (non-critical) |
|
||||
| Quarterly failover/failback test performed | Scheduled, not done | Yes |
|
||||
| One database restore test performed | Scheduled, not done | Yes |
|
||||
| One Docker volume restore test performed | Scheduled, not done | Yes |
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
# Gateway Lifecycle Troubleshooting
|
||||
|
||||
Diagnostic flow when `hermes gateway restart` doesn't work or the gateway won't start/stop correctly on a headless VPS (netcup/Hetzner, Debian).
|
||||
|
||||
## Symptom: `hermes gateway restart` fails with "linger is not enabled"
|
||||
|
||||
```
|
||||
Cannot restart gateway as a service -- linger is not enabled.
|
||||
The gateway user service requires linger to function on headless servers.
|
||||
Run: sudo loginctl enable-linger root
|
||||
```
|
||||
|
||||
On containers, LXC, or VPS environments without a DBus session bus, this will fail with:
|
||||
|
||||
```
|
||||
Failed to connect to system scope bus via local transport: Connection refused
|
||||
```
|
||||
|
||||
On these systems the gateway may run via a **system-level** `hermes.service` unit (at `/etc/systemd/system/`), not a user-level one. The user-level service guard (`linger`) is a red herring -- the system-level unit uses `User=root` and the global `EnvironmentFile`, bypassing the DBus session issue entirely.
|
||||
|
||||
## Diagnostic Steps
|
||||
|
||||
### 1. Find what's running
|
||||
|
||||
```bash
|
||||
# Check system-level service
|
||||
systemctl cat hermes.service # shows the full unit
|
||||
systemctl status hermes.service # active, failed, auto-restart loop?
|
||||
|
||||
# Check for manually-started gateway processes
|
||||
ps aux | grep "hermes.*gateway" | grep -v grep
|
||||
```
|
||||
|
||||
This reveals the critical fork: is the gateway running under systemd or as an orphan process?
|
||||
|
||||
### 2. Categorize what you find
|
||||
|
||||
| Situation | What to do |
|
||||
|-----------|-----------|
|
||||
| Gateway running under systemd (`hermes.service` active) | `systemctl restart hermes.service` or `hermes gateway restart` if the unit is not user-level |
|
||||
| Gateway running as manual process (no systemd wrapper, PID started with `hermes gateway run`) | `hermes gateway stop` kills it; then systemd's `Restart=on-failure` will pick it up cleanly |
|
||||
| `hermes.service` in `auto-restart` loop (exit-code 1) | Check journal: `journalctl -u hermes.service -n 10`. Often the cause is "Gateway already running" -- the manual process blocks the systemd one. Kill the manual process. |
|
||||
| No gateway at all | `systemctl start hermes.service` or `hermes gateway run` |
|
||||
|
||||
### 3. The common cause: dual processes
|
||||
|
||||
The gateway can be started in two ways:
|
||||
- **Manually:** `hermes gateway run` (or `hermes gateway run --replace`)
|
||||
- **As a systemd service:** `hermes.service` at `/etc/systemd/system/`
|
||||
|
||||
If someone starts it manually (e.g. `hermes gateway run` in a tmux session), systemd's `hermes.service` will fail with "Gateway already running" and loop forever. The fix:
|
||||
|
||||
```bash
|
||||
hermes gateway stop # kills the manual process
|
||||
# systemd auto-restarts within 5s -- check:
|
||||
sleep 6 && systemctl is-active hermes.service
|
||||
```
|
||||
|
||||
### 4. Stale systemd unit warning
|
||||
|
||||
After restarting, check logs for:
|
||||
|
||||
```
|
||||
Stale systemd unit detected: hermes.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain.
|
||||
```
|
||||
|
||||
This means the systemd unit was installed or regenerated at a different drain timeout than the current config. Fix:
|
||||
|
||||
```bash
|
||||
hermes gateway install --force
|
||||
```
|
||||
|
||||
This rewrites the unit with `TimeoutStopSec = drain_timeout + 30s`.
|
||||
|
||||
### 5. User-level vs system-level units
|
||||
|
||||
This environment has BOTH:
|
||||
- **System-level:** `/etc/systemd/system/hermes.service` -- runs the main Hermes gateway (default profile). `User=root`, uses `EnvironmentFile=/root/.hermes/.env`. This is the primary unit.
|
||||
- **User-level:** `~/.config/systemd/user/hermes-gateway-anita.service` -- runs the Anita profile gateway. Requires `loginctl enable-linger` (on this VPS, the system bus is unavailable so user-level units can't start).
|
||||
|
||||
Only the system-level unit matters for the default profile gateway. The user-level unit is for the Anita profile only.
|
||||
|
||||
### 6. Profile gateways
|
||||
|
||||
The Anita profile runs as a separate systemd service. Check its status:
|
||||
|
||||
```bash
|
||||
cat /proc/<pid>/cmdline | tr '\0' ' ' # tells you which profile is running
|
||||
ps aux | grep "profile anita" # Anita's gateway process
|
||||
```
|
||||
|
||||
Multiple gateway processes (one per profile) can run simultaneously without conflict.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# See the unit definition
|
||||
systemctl cat hermes.service
|
||||
|
||||
# Check gateway status
|
||||
hermes gateway status
|
||||
|
||||
# Kill manual process, let systemd take over
|
||||
hermes gateway stop
|
||||
|
||||
# Regenerate the systemd unit with correct timeouts
|
||||
hermes gateway install --force
|
||||
|
||||
# Check latest logs
|
||||
journalctl -u hermes.service --no-pager -n 15
|
||||
|
||||
# Check for any gateway-related processes
|
||||
ps aux | grep "hermes.*gateway" | grep -v grep
|
||||
# Look for 'hermes gateway run' vs 'python -m hermes_cli.main gateway run'
|
||||
# The first is manual, the second is often tmux/systemd-launched
|
||||
```
|
||||
@@ -0,0 +1,49 @@
|
||||
# Hermes Memory Auto-Consolidation
|
||||
|
||||
**Deployed:** July 9, 2026
|
||||
**Schedule:** Every 10 min (Hermes cron, no_agent mode)
|
||||
**Scripts:** `/root/.hermes/scripts/hermes-consolidate.sh` + `hermes-consolidate.py`
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Every 10 min:
|
||||
1. Dump MEMORY.md entries to JSON
|
||||
2. Upload to S3 (backup-before-prune guarantee — aborts if S3 fails)
|
||||
3. Run pruner (pattern prune + size safety valve)
|
||||
4. Prune fact store (entries older than 7 days, never-retrieved, trust < 0.5)
|
||||
5. Keep local pre-prune copy in ~/.hermes/memories/.consolidate-backups/ (48h retention)
|
||||
```
|
||||
|
||||
## Safety Guarantees
|
||||
|
||||
- **Backup-before-prune:** S3 upload is a hard gate; prune never runs without it
|
||||
- **Never writes empty file:** pruner aborts if kept set would be empty
|
||||
- **Atomic write:** `os.replace()` on temp file — no partial-write risk
|
||||
- **Protected entries:** regex PROTECT prevents identity, rules, credentials, and key infrastructure facts from EVER being pruned (pattern or size valve)
|
||||
- **Idempotent:** if already under target, backs up and exits without modifying
|
||||
|
||||
## PROTECT List (entries never pruned)
|
||||
|
||||
As of Jul 9 2026: includes "Core:", "STANDING PRACTICE", "app1-bu", "netcup|admin-ai.itpropartner", "Ops portal", "Recovery manual", "Ollama", "RS 4000", "Migration target", plus all credential/identity/formatting rules.
|
||||
|
||||
## Restore Procedure
|
||||
|
||||
```bash
|
||||
# From S3:
|
||||
source /opt/awscli-venv/bin/activate
|
||||
aws s3 cp s3://hermes-vps-backups/live/memories/memory-backup.json /tmp/mem.json \
|
||||
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
|
||||
# Rebuild MEMORY.md:
|
||||
python3 -c "
|
||||
import json
|
||||
d = json.load(open('/tmp/mem.json'))
|
||||
open('/root/.hermes/memories/MEMORY.md','w').write(('\n§\n'.join(d['entries'])) + '\n')
|
||||
print('restored', d['entry_count'], 'entries')
|
||||
"
|
||||
```
|
||||
|
||||
## Known Caveat
|
||||
|
||||
The size safety valve drops oldest-appended entries when pattern-pruning alone doesn't reach 7,000 chars. This can remove useful entries at the top of the file if they don't match a PROTECT pattern. All data preserved in S3.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Server Pricing Comparison (Jul 2026)
|
||||
|
||||
## netcup Root Server G12 (current)
|
||||
Monthly pricing (incl. 0% VAT). Dedicated AMD EPYC 9645 cores.
|
||||
|
||||
| Plan | Cores | RAM | NVMe | EUR/mo | USD/mo |
|
||||
|------|-------|-----|------|--------|--------|
|
||||
| RS 1000 | 4 | 8 GB | 256 GB | €10.74 | ~$12.24 |
|
||||
| RS 2000 ← Core | 8 | 16 GB | 512 GB | €18.00 | ~$20.52 |
|
||||
| RS 4000 | 12 | 32 GB | 1 TB | €33.54 | ~$38.24 |
|
||||
|
||||
## Hetzner Cloud (current — inflated pricing)
|
||||
|
||||
| Tier | vCPU | RAM | Disk | EUR/mo | USD/mo |
|
||||
|------|------|-----|------|--------|--------|
|
||||
| CPX11 | 2 | 2 GB | 40 GB | ~€11 | ~$13 |
|
||||
| CPX21 | 3 | 4 GB | 80 GB | ~€32 | ~$36 |
|
||||
| CPX32 | 4 | 8 GB | 160 GB | ~€35 | ~$40 |
|
||||
| CPX42 | 8 | 16 GB | 320 GB | ~€69 | ~$79 |
|
||||
|
||||
## Key Insight
|
||||
Hetzner's pricing was raised significantly for new/recreated servers as of Jul 2026. The API returns inflated rates. Core's equivalent compute (8C/16G) at Hetzner would be CPX42 at ~$79/mo vs **€18/mo (~$20.52)** on netcup RS 2000 — a ~74% savings.
|
||||
|
||||
**Strategy:** Consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers. Freed-up existing Hetzner boxes can be reused for standby duty without incurring the inflated pricing.
|
||||
|
||||
## Exchange Rate
|
||||
€1 ≈ $1.14 USD (as of Jul 8, 2026)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,24 @@
|
||||
# Hermes Memory Auto-Consolidation — DR / Restore Procedure
|
||||
|
||||
**Component:** `hermes-consolidate.sh` + `hermes-consolidate.py`
|
||||
**Schedule:** every 10 min (Hermes cron, `no_agent`)
|
||||
**Last reviewed:** July 9, 2026 — Network Services Team (Backup Specialist)
|
||||
|
||||
---
|
||||
|
||||
## What it does
|
||||
|
||||
Every 10 minutes:
|
||||
1. Dumps MEMORY.md to JSON
|
||||
2. Uploads backup to S3 FIRST (aborts prune on failure)
|
||||
3. Prunes stale entries
|
||||
4. Keeps local timestamped copies for 48h
|
||||
|
||||
## Safety guarantees
|
||||
- Backup-before-prune: upload to S3 is a hard gate
|
||||
- Never writes empty file
|
||||
- Atomic write via os.replace()
|
||||
- Protected entries never pruned
|
||||
|
||||
## Restore procedures
|
||||
See full document at /root/.hermes/cache/documents/doc_498756c88128_memory-consolidation-dr-sanitized.md
|
||||
@@ -0,0 +1,43 @@
|
||||
# Formatting rules for user-facing reference files
|
||||
|
||||
These rules apply to all `.md` files the user may view on mobile (Telegram preview, file open, email attachment).
|
||||
|
||||
## ASCII-only status indicators
|
||||
|
||||
Do NOT use emoji for status indicators in files the user reads. Common mobile viewers misinterpret UTF-8 emoji sequences as Latin-1 characters, producing garbled text like `🔴` instead of `:red_circle:`.
|
||||
|
||||
Use these replacements instead:
|
||||
|
||||
- :red_circle: -> [HIGH]
|
||||
- :yellow_circle: -> [MED] or [WARN]
|
||||
- :green_circle: -> [OK] or [LOW]
|
||||
- :check_mark_button: -> [OK]
|
||||
- :check_mark: -> [OK]
|
||||
- :cross_mark: -> [FAIL]
|
||||
- :globe_with_meridians: -> [INFO]
|
||||
- :cyclone: -> [PENDING]
|
||||
- :right_arrow: -> ->
|
||||
- em dash -- -> --
|
||||
- times sign x -> x
|
||||
|
||||
## No pipe tables in summaries
|
||||
|
||||
Markdown pipe tables (`| col | col |`) are unreadable on small phone screens. Use bullet lists or comma-separated inline format instead:
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
| ID | Issue | Status |
|
||||
|----|-------|--------|
|
||||
| 1 | Caddyfile | Fixed |
|
||||
```
|
||||
|
||||
**Good:**
|
||||
```
|
||||
Fixed:
|
||||
- 1 Caddyfile -> Fixed
|
||||
- 2 Backups -> Fixed
|
||||
```
|
||||
|
||||
## Keep it pure ASCII
|
||||
|
||||
Any non-ASCII character is a risk. Stick to `[0-9A-Za-z]`, hyphens, underscores, periods, and standard punctuation. Smart quotes, long dashes, mathematical symbols, and Unicode arrows all break on some mobile renderers.
|
||||
@@ -0,0 +1,37 @@
|
||||
# MSP Documentation Suite
|
||||
|
||||
Generated Jul 10, 2026. Standardized document structure for the IT Pro Partner MSP.
|
||||
|
||||
## Core Infrastructure Docs
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| DR Plan v3 | /root/.hermes/references/server-dr-plans-v3.md | RTO/RPO targets, failover architecture, provider diversity |
|
||||
| DR Plan Public | /root/.hermes/references/server-dr-plans-public.md | Redacted for Reddit/Facebook |
|
||||
| Server Inventory | /root/.hermes/references/server-inventory.md | All 9 servers: specs, IPs, roles, costs, status |
|
||||
| Application Inventory | /root/.hermes/references/application-inventory.md | Every service: ports, dependencies, backup methods |
|
||||
| Server Provisioning Standard | /root/.hermes/references/server-provisioning-standard-public.md | Step-by-step build checklist |
|
||||
| Network Diagram | /root/.hermes/references/network-diagram.md | Topology, ports, VPNs, DNS zones |
|
||||
|
||||
## Security & Recovery
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| Backup Policy | /root/.hermes/references/backup-policy.md | What/when/how, retention, RPO, verification |
|
||||
| Incident Response Plan | /root/.hermes/references/incident-response-plan.md | Severity levels, team, comms, escalation |
|
||||
| Restore Runbooks | /root/.hermes/references/restore-runbooks.md | Per-server numbered disaster recovery steps |
|
||||
| DR Issue Log | /root/.hermes/references/dr-issue-log.md | Permanent record of all findings and fixes |
|
||||
|
||||
## Operations
|
||||
|
||||
| Document | Path | Purpose |
|
||||
|---|---|---|
|
||||
| CloudPanel vs Enhance vs Coolify | /root/runcloud_replacements_report.md | WordPress panel selection research |
|
||||
| Model Price Research | /root/.hermes/references/model-price-research.md | LLM pricing comparison across providers |
|
||||
|
||||
## Gaps Still Needed
|
||||
|
||||
- Client Onboarding Checklist
|
||||
- Service Catalog (what ITPP offers, pricing tiers)
|
||||
- Access Control Policy
|
||||
- Change Management Log
|
||||
@@ -0,0 +1,80 @@
|
||||
# Netcup SCP REST API Access
|
||||
|
||||
## Authentication
|
||||
|
||||
The netcup SCP API at `servercontrolpanel.de` uses **Keycloak OIDC** — NOT API key headers. The CCP API key from `customercontrolpanel.de` Master Data does **not** work with the SCP REST API directly.
|
||||
|
||||
### Get a token
|
||||
|
||||
```bash
|
||||
TOKEN=$(curl -s "https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token" \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=password&client_id=scp&username=${NETCUP_CUSTOMER}&password=${NETCUP_PASSWORD}&scope=openid" \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token')")
|
||||
```
|
||||
|
||||
| Parameter | Value | Source |
|
||||
|-----------|-------|--------|
|
||||
| Token endpoint | `https://www.servercontrolpanel.de/realms/scp/protocol/openid-connect/token` | From `keycloak.js` in SCP UI |
|
||||
| Client ID | `scp` | From `keycloak.js` |
|
||||
| Grant type | `password` | Resource owner password grant |
|
||||
| Username | CCP customer number (numeric, e.g. `389212`) | From CCP Master Data |
|
||||
| Password | CCP account password | User-provided |
|
||||
| Scope | `openid` | Required for ID token |
|
||||
| Token lifetime | 300 seconds (5 min) | From token response `expires_in` |
|
||||
| Refresh | 1800 seconds (30 min) | From token response `refresh_expires_in` |
|
||||
|
||||
### API base URL
|
||||
|
||||
The SCP has three API paths:
|
||||
|
||||
| Path | Purpose | Status |
|
||||
|------|---------|--------|
|
||||
| `https://www.servercontrolpanel.de/scp-core/api/v1/` | Server management CRUD | ✅ Works (returns JSON) |
|
||||
| `https://www.servercontrolpanel.de/scp-agent/api/v1/` | Agent/service operations | ⚠️ Returns 403 Forbidden |
|
||||
| `https://www.servercontrolpanel.de/scp-ui/api/v1/` | UI data endpoints | ❌ Returns HTML login page, not JSON |
|
||||
|
||||
**Always use `scp-core` for server operations.**
|
||||
|
||||
## Available endpoints
|
||||
|
||||
### List servers
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/servers" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
Returns: `[{"id": 890903, "name": "v2202607377162478911", "template": {"name": "RS 2000 G12"}}]`
|
||||
|
||||
### Single server details
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/servers/890903" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
|
||||
Returns: IP addresses, hostname, architecture, template name, disk available space, RAM.
|
||||
|
||||
### Templates / Images (provisioning)
|
||||
```bash
|
||||
curl -s "https://www.servercontrolpanel.de/scp-core/api/v1/templates" \
|
||||
-H "Authorization: Bearer ${TOKEN}"
|
||||
```
|
||||
**Note:** May return `{"code": "error.notfound", "message": "Resource not found"}` for non-reseller accounts. This is a known limitation — provisioning via API may require reseller status with netcup.
|
||||
|
||||
## Credentials
|
||||
|
||||
Stored in `/root/.hermes/.env`:
|
||||
```
|
||||
NETCUP_CUSTOMER=389212
|
||||
NETCUP_PASSWORD=<CCP password>
|
||||
NETCUP_KEY=<API key from Master Data page>
|
||||
NETCUP_API_KEY=<same as NETCUP_KEY>
|
||||
```
|
||||
|
||||
The API key from the Master Data page is NOT used by the REST API. The password grant flows use the customer number + CCP password directly. The API key exists for legacy SOAP/JSON-RPC endpoints.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- **Token expires every 5 minutes** — need to re-authenticate or implement refresh flow for long-running operations
|
||||
- **Provisioning may be restricted** — `/templates` and `/images` return 404 for non-reseller accounts
|
||||
- **This is the SCP (Server Control Panel), not CCP** — the CCP (`customercontrolpanel.de`) has a separate API surface for billing/invoicing
|
||||
- **Discovered Jul 8 2026** — after multiple failed attempts with API key auth that returned 404/401, the working auth flow was found by inspecting `keycloak.js` in the SCP UI JavaScript bundle
|
||||
@@ -0,0 +1,70 @@
|
||||
# IT Pro Partner — Network & Service Endpoints
|
||||
|
||||
## DNS & Hosting
|
||||
|
||||
| Domain | Nameservers | Hosting |
|
||||
|--------|------------|---------|
|
||||
| itpropartner.com | SiteGround (ns1/ns2.siteground.net) | SiteGround |
|
||||
| iamgmb.com | Cloudflare | MXroute (email) |
|
||||
| germainebrown.com | Cloudflare | MXroute (email) |
|
||||
| debtrecoveryexperts.com | Cloudflare | SiteGround/self-hosted |
|
||||
|
||||
## Core (netcup RS 2000 G12)
|
||||
|
||||
| Service | Port | URL / Notes |
|
||||
|---------|------|-------------|
|
||||
| Hermes gateway | — | Telegram bot |
|
||||
| Hermes assistant (PWA) | 8082 | app.itpropartner.com |
|
||||
| Shark game API | 8083 | shark.iamgmb.com |
|
||||
| Chromium CDP | 9222 | local browser automation |
|
||||
| SMTP relay | 2525 | mail.germainebrown.com (STARTTLS) |
|
||||
| Caddy | 443 | Reverse proxy for all sites |
|
||||
| Tailscale | — | vaultwarden.tailc2f3b0.ts.net |
|
||||
|
||||
## Hetzner Servers
|
||||
|
||||
| Name | Type | IP | Purpose |
|
||||
|------|------|----|---------|
|
||||
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
|
||||
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby |
|
||||
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
|
||||
| unms | CPX21 | 5.161.225.131 | UISP/UNMS |
|
||||
| unifi | CPX21 | 178.156.131.57 | UniFi controller |
|
||||
| hudu | CPX21 | 178.156.130.130 | IT documentation |
|
||||
| ai | CPX41 | 178.156.167.181 | admin-ai LLM proxy |
|
||||
| fleet | CPX11 | 178.156.149.32 | Fleet tracker |
|
||||
| docker | CPX11 | 178.156.168.35 | Docker host |
|
||||
| app1 (old) | CPX11 | 87.99.144.163 | Legacy |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| API | Auth | Base URL |
|
||||
|-----|------|----------|
|
||||
| Hetzner Cloud | Bearer token | https://api.hetzner.cloud/v1 |
|
||||
| Cloudflare | Bearer token | https://api.cloudflare.com/client/v4 |
|
||||
| admin-ai | Bearer token | https://admin-ai.itpropartner.com/v1 |
|
||||
| netcup SCP | Keycloak OIDC | https://www.servercontrolpanel.de/scp-core/api/v1 |
|
||||
| Firecrawl | Bearer token | https://api.firecrawl.com/v1 |
|
||||
| Wasabi S3 | AWS4-HMAC-SHA256 | s3.wasabisys.com |
|
||||
|
||||
## S3 Backup Buckets
|
||||
|
||||
| Bucket | Prefix | Purpose |
|
||||
|--------|--------|---------|
|
||||
| hermes-vps-backups | live/ | Active Hermes state (15 min sync) |
|
||||
| hermes-vps-backups | standby/ | Standby recovery bundle |
|
||||
| hermes-vps-backups | hermes-full-backup/ | Daily 5 AM full tarball |
|
||||
| itpropartner-backups | — | Portal files & public data |
|
||||
| mikrotik-ccr-backups | wisp-backups/ | Router configs |
|
||||
|
||||
## External Services
|
||||
|
||||
| Service | Login URL | Notes |
|
||||
|---------|-----------|-------|
|
||||
| netcup CCP | customercontrolpanel.de | Account management |
|
||||
| netcup SCP | servercontrolpanel.de | Server management |
|
||||
| Hetzner Cloud | console.hetzner.cloud | Hetzner project |
|
||||
| Cloudflare | dash.cloudflare.com | DNS, Access, Registrar |
|
||||
| MXroute | mxlogin.com | Email hosting |
|
||||
| SiteGround | siteground.com | WP hosting |
|
||||
| Wasabi | console.wasabisys.com | S3-compatible storage |
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
# Push Notification Pitfalls (iOS PWA)
|
||||
|
||||
Captured from the Jul 9, 2026 shark game push notification debugging session.
|
||||
|
||||
## iOS Safari does NOT support PushManager in browser tabs
|
||||
|
||||
On iPhones, `PushManager` is only available when the web app is opened from the Home Screen as a standalone PWA (no URL bar, no browser chrome). Safari's browser tab does NOT expose it.
|
||||
|
||||
**Symptom:** `FAIL: No PushManager` when tapping "Enable Notifications" in Safari.
|
||||
|
||||
**Fix:** User must:
|
||||
1. Tap Share icon -> "Add to Home Screen"
|
||||
2. Open from home screen icon (fullscreen mode)
|
||||
3. Push subscription now works via Apple Push Service (APNs endpoint: `web.push.apple.com`)
|
||||
|
||||
## Subscription to APNs requires the subscribe-noauth endpoint
|
||||
|
||||
The standard `POST /api/push/subscribe` requires JWT authentication. For test pages without login, create a separate `POST /api/push/subscribe-noauth` endpoint that stores subscriptions with `user_id: NULL` to avoid FK constraint failures.
|
||||
|
||||
## Database schema: user_id must be nullable
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS push_subscriptions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
user_id INTEGER, -- NULL for no-auth subscriptions; FK to users.id otherwise
|
||||
endpoint TEXT NOT NULL UNIQUE,
|
||||
p256dh TEXT NOT NULL,
|
||||
auth TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
```
|
||||
|
||||
## Common backend errors
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `FOREIGN KEY constraint failed` | user_id=0 but no user.id=0 exists | Use NULL or valid FK |
|
||||
| `{"sent":0}` | All push send attempts failed silently | Change `except: pass` to `except: print(f"Push failed: {e}")` |
|
||||
| `name 'json' is not defined` | `import json` missing at top of server.py | Add `import json` to imports |
|
||||
| `ModuleNotFoundError: pywebpush` | Installed in wrong venv | Install in the systemd service's venv, not system Python |
|
||||
@@ -0,0 +1,31 @@
|
||||
# IT Pro Partner — Per-Server Disaster Recovery Plan (REDACTED)
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team
|
||||
**Date:** July 9, 2026
|
||||
**Version:** 1.0 — Redacted for 3rd Party Review
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
11 servers total. 1 primary (dedicated EPYC), 10 Hetzner cloud VPSes.
|
||||
1 warm standby. Shared dependencies: S3-compatible storage, Cloudflare DNS, SSH key auth.
|
||||
|
||||
## DR Classification
|
||||
|
||||
| Tier | Count | RTO | RPO |
|
||||
|------|-------|-----|-----|
|
||||
| Critical | 2 | 5 min - 2 hours | 15 min - 24 hours |
|
||||
| Important | 4 | 4 hours | 24 hours |
|
||||
| Standard | 4 | 8 hours | 24 hours |
|
||||
| Standby | 1 | 2 min | 15 min |
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. AI/LLM server disk at 92% — cleanup needed before migration
|
||||
2. Docker volume backups not automated to S3
|
||||
3. Database dumps not scheduled for most servers
|
||||
4. Hermes warm standby functional via S3 sync + WireGuard tunnel
|
||||
|
||||
## Full version
|
||||
Full DR plans with IPs, credentials, and detailed restore procedures are stored internally at /root/.hermes/references/server-dr-plans.md and available on request.
|
||||
@@ -0,0 +1,292 @@
|
||||
# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
|
||||
**Date:** July 10, 2026
|
||||
**Status:** LIVE — replaces v1.0
|
||||
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. Architecture & Shared Dependencies
|
||||
2. Server Standards
|
||||
3. Backup Strategy
|
||||
4. Failover / Failback
|
||||
5. Monitoring & Alerting
|
||||
6. Restore Runbooks
|
||||
7. Testing & Validation
|
||||
8. Security
|
||||
9. Immediate Action Items
|
||||
10. Appendix: Third-Party Review Responses
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture & Shared Dependencies
|
||||
|
||||
```
|
||||
Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
|
||||
|
|
||||
+-----------+-----------+
|
||||
| | |
|
||||
SSH/WG Tailscale Management
|
||||
(itpp- (Core <-> VPN (WireGuard
|
||||
infra) app1-bu) to home router)
|
||||
| | |
|
||||
+-----------+-----------+
|
||||
|
|
||||
+-----------+-----------+
|
||||
| |
|
||||
S3 Primary S3 Secondary
|
||||
(Wasabi us-east-1) (Wasabi us-west-2
|
||||
versioning ON or Backblaze B2)
|
||||
object lock ON object lock ON
|
||||
```
|
||||
|
||||
### Shared Dependencies
|
||||
|
||||
| Dependency | Primary | Backup / Fallback | Credentials Location |
|
||||
|---|---|---|---|
|
||||
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
|
||||
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
|
||||
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
|
||||
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
|
||||
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
|
||||
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
|
||||
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
|
||||
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
|
||||
|
||||
---
|
||||
|
||||
## 2. Server Standards
|
||||
|
||||
### 2.1 Tier Definitions
|
||||
|
||||
| Tier | Spec | OS | Use | Cost |
|
||||
|---|---|---|---|---|
|
||||
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
|
||||
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
|
||||
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
|
||||
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |
|
||||
|
||||
### 2.2 Base Install -- Verified Checklist
|
||||
|
||||
```bash
|
||||
# S1: hostname + timezone
|
||||
hostnamectl set-hostname <name>
|
||||
timedatectl set-timezone America/New_York
|
||||
|
||||
# S2: system update
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# S3: security baseline
|
||||
apt install -y fail2ban ufw
|
||||
ufw default deny incoming; ufw default allow outgoing
|
||||
ufw allow ssh; ufw --force enable
|
||||
|
||||
# S4: monitoring
|
||||
apt install -y prometheus-node-exporter
|
||||
|
||||
# S5: standard user + key
|
||||
adduser ippadmin && usermod -aG sudo ippadmin
|
||||
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
|
||||
|
||||
# S6: harden SSH
|
||||
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
systemctl restart sshd
|
||||
|
||||
# S7: Docker (if needed)
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin
|
||||
```
|
||||
|
||||
### 2.3 Application Data Paths
|
||||
|
||||
Every server must document:
|
||||
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
|
||||
- What can be rebuilt vs what must be restored
|
||||
- Secrets location
|
||||
- Caddy/nginx configs
|
||||
- Cron jobs
|
||||
- External API dependencies
|
||||
|
||||
---
|
||||
|
||||
## 3. Backup Strategy
|
||||
|
||||
### 3.1 Backup Matrix
|
||||
|
||||
| Data Type | Frequency | Target | Retention | Status |
|
||||
|---|---|---|---|---|
|
||||
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
|
||||
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
|
||||
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE |
|
||||
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE |
|
||||
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
|
||||
| System configs | Daily | S3 primary (<server>/) | 90 days | 🔲 NOT AUTOMATED |
|
||||
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Application data | Per-app | S3 primary (<server>/) | 90 days | 🔲 PARTIAL |
|
||||
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |
|
||||
|
||||
### 3.2 Backup Security
|
||||
|
||||
- **Versioning:** ON (all 3 buckets)
|
||||
- **Object Lock:** NOT enabled
|
||||
- **Delete protection:** NOT configured
|
||||
- **Backup credentials:** Admin-level (can delete) -- NOT write-only
|
||||
- **Encryption:** Server-side only (Wasabi default)
|
||||
|
||||
---
|
||||
|
||||
## 4. Failover / Failback
|
||||
|
||||
### 4.1 Warm Standby Timing (Corrected)
|
||||
|
||||
**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)
|
||||
|
||||
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO
|
||||
|
||||
### 4.2 Health Check -- Two Layers
|
||||
|
||||
**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
|
||||
**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama
|
||||
|
||||
### 4.3 Split-Brain Prevention
|
||||
|
||||
S3 active-lock.json mechanism:
|
||||
|
||||
```json
|
||||
{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}
|
||||
```
|
||||
|
||||
Core writes heartbeat every 30s while healthy. Failover only proceeds after:
|
||||
1. Core unreachable for 4 consecutive checks (app1-bu local)
|
||||
2. External monitor independently confirms Core down
|
||||
3. active-lock.json heartbeat stale (>90s)
|
||||
4. app1-bu writes itself as lock_holder
|
||||
5. Gateway starts
|
||||
6. Operator alerted
|
||||
|
||||
If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.
|
||||
|
||||
### 4.4 Failback Protocol
|
||||
|
||||
**No auto-failback.** Sequence:
|
||||
1. Confirm app1-bu is active node
|
||||
2. Keep Core stopped -- no auto-start
|
||||
3. Restore Core OS + stack
|
||||
4. Sync latest state from app1-bu/S3 into Core
|
||||
5. Validate sync timestamp
|
||||
6. Start Core in passive mode
|
||||
7. Stop app1-bu gateway
|
||||
8. Update active-lock.json: "active_node": "core"
|
||||
9. Start Core gateway
|
||||
10. 5 consecutive health checks pass
|
||||
11. Release standby lock
|
||||
12. Monitor 30 min before closing
|
||||
|
||||
**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.
|
||||
|
||||
### 4.5 Provider/Account Outage
|
||||
|
||||
If entire netcup account is unavailable:
|
||||
1. app1-bu (Hetzner) takes over
|
||||
2. DNS records pointed to app1-bu IP
|
||||
3. No auto-failback -- human decision
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-Server RTO/RPO Targets
|
||||
|
||||
| System | RTO | RPO | Notes |
|
||||
|---|---|---|---|
|
||||
| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) |
|
||||
| app1-bu (standby) | N/A | <=15 min | Recovery path for Core |
|
||||
| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose |
|
||||
| wphost02 (WordPress) | 2h | 24h | DB restore from dump |
|
||||
| fleettracker360 | 2h | 24h | DB restore from dump |
|
||||
| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Immediate Action Items
|
||||
|
||||
### P0 -- This Week
|
||||
|
||||
| # | Action | Details |
|
||||
|---|---|---|
|
||||
| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs |
|
||||
| 2 | Create S3 write-only IAM user | PutObject only, no delete/list |
|
||||
| 3 | Enable S3 Object Lock | 7-day minimum retention |
|
||||
| 4 | Deploy Docker volume backup | All Docker hosts |
|
||||
| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) |
|
||||
| 6 | Deploy 30s watchdog | Currently documented, need deployment |
|
||||
|
||||
### P1 -- Next 2 Weeks
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 7 | Document Tony's Hermes config + backup to S3 |
|
||||
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) |
|
||||
| 9 | Extend snapshot retention from 48h to 30 days |
|
||||
| 10 | Add backup success/failure monitoring to ops portal |
|
||||
|
||||
### P2 -- Next 30 Days
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 11 | Full DR simulation for Core failover |
|
||||
| 12 | Monthly restore testing schedule begins |
|
||||
| 13 | Deploy Prometheus + Grafana stack |
|
||||
| 14 | Add disk threshold alerts for all servers |
|
||||
| 15 | Begin netcup migration |
|
||||
|
||||
---
|
||||
|
||||
## 7. Restore Runbooks (Summary)
|
||||
|
||||
Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:
|
||||
- Purpose + dependencies
|
||||
- DNS records, firewall ports, backup path
|
||||
- Required secrets
|
||||
- Step-by-step restore commands
|
||||
- DB restore with pre-import dump verification
|
||||
- Docker volume restore
|
||||
- Health checks
|
||||
- Known failure modes
|
||||
|
||||
Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.
|
||||
|
||||
---
|
||||
|
||||
## 8. Third-Party Review Responses
|
||||
|
||||
### Reviewer 1 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks |
|
||||
| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan |
|
||||
| AI server 92% disk critical | Section 6 -- P0 action item #1 |
|
||||
| Memory consolidation data loss risk | Separate docs per priority-tagging proposal |
|
||||
| Tony's Hermes undocumented | Section 6 -- P1 action item #7 |
|
||||
|
||||
### Reviewer 2 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan |
|
||||
| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type |
|
||||
| Restore procedures not detailed | Section 7 -- Full runbooks reference |
|
||||
| Failback under-defined | Section 4.4 -- Protocol |
|
||||
| No backup restore testing | Separate testing schedule doc |
|
||||
| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 |
|
||||
| Backup security not described | Section 3.2 |
|
||||
| Application backup details vague | Section 2.3 |
|
||||
| Secrets recovery not documented | Section 1 (shared deps table) |
|
||||
| Provider/account outage missing | Section 4.5 |
|
||||
|
||||
---
|
||||
|
||||
*This plan is ACTIVE. Review quarterly or after any infrastructure change.*
|
||||
@@ -0,0 +1,38 @@
|
||||
# IT Pro Partner — Per-Server Disaster Recovery Plan
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team
|
||||
**Date:** July 9, 2026
|
||||
**Version:** 1.0
|
||||
**Stored at:** /root/.hermes/references/server-dr-plans.md (primary), s3://hermes-vps-backups/live/config/server-dr-plans.md (S3 backup)
|
||||
|
||||
**Redacted version:** /root/.hermes/references/server-dr-plans-redacted.md (for 3rd party review)
|
||||
|
||||
---
|
||||
|
||||
## Server Inventory
|
||||
|
||||
| # | Server | IP | Provider | Type | Status | Role |
|
||||
|---|--------|-----|----------|------|--------|------|
|
||||
| 1 | core | 152.53.192.33 | netcup | RS 2000 G12 | 🟢 LIVE | Primary ops |
|
||||
| 2 | wphost02 | 5.161.62.38 | Hetzner | CPX21 | 🟢 LIVE | WordPress/RunCloud |
|
||||
| 3 | ai | 178.156.167.181 | Hetzner | CPX41 | 🟡 LIVE (92% disk) | LiteLLM + Ollama |
|
||||
| 4 | unms | 5.161.225.131 | Hetzner | CPX21 | 🟢 LIVE | UISP/UNMS |
|
||||
| 5 | unifi | 178.156.131.57 | Hetzner | CPX21 | 🟢 LIVE | UniFi Controller |
|
||||
| 6 | hudu | 178.156.130.130 | Hetzner | CPX21 | 🟢 LIVE | IT documentation |
|
||||
| 7 | fleettracker360 | 178.156.149.32 | Hetzner | CPX11 | 🟢 LIVE | Fleet tracking |
|
||||
| 8 | docker | 178.156.168.35 | Hetzner | CPX11 | 🟢 LIVE | Utility Docker |
|
||||
| 9 | n8n | 87.99.144.163 | Hetzner | CPX11 | 🟢 LIVE | Automation |
|
||||
| 10 | tony-vps | 87.99.159.142 | Hetzner | CPX21 | 🟢 LIVE | Tony's Hermes |
|
||||
| 11 | app1-bu | 5.161.114.8 | Hetzner | CPX11 | 🟢 STANDBY | Warm standby |
|
||||
|
||||
## Full DR plans
|
||||
See the full document at /root/.hermes/references/server-dr-plans.md for:
|
||||
- Per-server: services, Docker containers, backup paths, restore procedures
|
||||
- DR classification matrix (RTO/RPO/Complexity per server)
|
||||
- Shared dependency map (SSH key, S3, SMTP, DNS, Cloudflare)
|
||||
- ai.itpropartner.com pre-reboot checklist (🔴 critical — 92% disk)
|
||||
- app1-bu failover procedure
|
||||
- Recovery bundle locations
|
||||
|
||||
## Redacted version
|
||||
See /root/.hermes/references/server-dr-plans-redacted.md for the sanitized version suitable for external review.
|
||||
Reference in New Issue
Block a user