ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

Browse Source
This commit is contained in:
root
2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
references/1-DR-Master-Plan-v2.md
+109
View File
@@ -0,0 +1,109 @@
IT PRO PARTNER — DOCUMENT 1 OF 4
Disaster Recovery Master Plan
v2.0 — Firm RTO/RPO targets, failover design, and failback procedure
Field
Value
Document status
ACTIVE — supersedes v1 draft and all prior DR notes
Date
July 9, 2026
Priority directive
Uptime is critical. Full-service restorability takes priority over cost or convenience.
Owner
Network Services Team
Companion documents
Backup & Security Hardening Standard; Per-Server Runbooks; DR Testing & Validation Schedule
1. Priority Directive
This plan is written under one governing rule: uptime and full-service restorability are the top priority, ahead of infrastructure cost, engineering convenience, or process overhead. Every target below is set to that standard — they are commitments, not estimates, and they drive the hardening work in the companion Backup & Security Hardening Standard.
2. RTO / RPO Targets — Committed
These are the recovery targets ITPP is designing and testing against. They replace the “not yet defined” placeholders from the v1 draft.
Tier / System
RTO (target)
RPO (target)
Basis
Core (Hermes)
5 minutes
15 minutes
Warm standby (app1-bu) + 15-min Hermes state sync
app1-bu (standby)
N/A — is the recovery path for Core
≤ 15 minutes (sync lag)
Continuous local state sync from Core
app1 / app2 / app3 (Standard tier)
60 minutes
24 hours
Rebuild from S3 + docker compose up, daily backups
wphost02 (WordPress)
2 hours
24 hours
Rebuild + DB restore, daily dumps
fleettracker360
2 hours
24 hours
Rebuild + DB restore, daily dumps
Legacy Hetzner hosts
4 hours
24 hours
Rebuild via rescue mode, daily backups
Why 5 minutes for Core, not less: A 2-minute RTO leaves no margin for S3 sync, service startup, DNS/cache propagation, or transient network issues. 5 minutes is achievable with the failover design in Section 3 and still leaves real margin. Do not compress this further without re-testing.
Compliance rule: These targets are only real once they are tested (see DR Testing & Validation Schedule). Until a target has passed a live test, treat it as “target, unverified,” not “achieved.”
3. Core Failover Design — app1 / app1-bu
3.1 Detection and trigger
Parameter
Value
Health check interval
30 seconds
Consecutive failed checks to trigger failover
4 (→ ~2 minutes detection time)
Check sources required to agree
app1-bu local check AND one external monitor
Startup buffer (service start + validation)
45–90 seconds
Resulting effective RTO
~3–3.5 minutes actual, 5 minutes committed target
A single missed check must never trigger failover — it has to clear 4 consecutive failures from two independent vantage points before app1-bu is allowed to act. This is what prevents a transient network blip on one path from causing an unnecessary takeover.
3.2 Split-brain prevention — required before go-live
Mechanism: an active/standby lock file maintained in S3, checked before any failover.
s3://hermes-vps-backups/live/active-lock.json
{ "active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core" }
Core writes a heartbeat to this object every 30 seconds while healthy. Failover sequence:
Confirm Core unreachable for 4 consecutive checks (local).
Confirm external monitor independently reports Core down.
Confirm the active-lock heartbeat is stale (older than 90 seconds).
app1-bu writes itself in as lock_holder and active_node.
app1-bu starts the Hermes gateway.
Alert the on-call operator immediately, success or failure.
Fencing: Where reachable, app1-bu should also attempt to firewall-isolate or force-stop Core before going active, to eliminate any chance both nodes serve traffic at once. If Core cannot be reached to fence, proceed on the lock file alone but flag the incident as high severity for manual review.
3.3 Failback procedure — authoritative steps
Failback, not failover, is where most DR programs actually lose data — typically because the restored primary starts serving before it has the standby's newer state. This sequence is mandatory and may not be shortcut during a live incident.
Confirm app1-bu is currently the authoritative active node (check active-lock.json).
Keep Core services stopped after it is repaired/restored — do not allow auto-start.
Restore Core's OS and application stack per its runbook.
Sync the latest state from app1-bu (or the authoritative S3 state) into Core.
Validate the synced state's timestamp is newer than Core's last known state before the outage.
Start Core in passive/maintenance mode only — not yet serving traffic.
Stop the Hermes gateway on app1-bu.
Update active-lock.json: active_node = core.
Start the Hermes gateway on Core.
Confirm Core health checks pass for at least 5 consecutive checks.
Release the standby lock; return app1-bu to passive standby mode.
Monitor both nodes for 30 minutes before closing the incident.
Rollback: If Core fails health checks after step 9, immediately revert: set active_node back to app1-bu in the lock file, restart the Hermes gateway on app1-bu, and keep Core in maintenance mode until root cause is found. Do not retry failback more than once without a human decision to proceed.
4. Standby Capacity Check
app1-bu runs on the Standby tier (Hetzner CPX11 — 2 vCPU / 2GB RAM / 40GB). Under the failover design above, it must be able to run the Hermes gateway plus health-check/lock tooling concurrently within that footprint.
Action required: Load-test app1-bu with the Hermes gateway running under representative traffic before relying on the 5-minute RTO commitment. If headroom is marginal, upgrade the standby tier rather than compress the failover design to fit — uptime takes priority over the ~$8/mo savings of staying on CPX11.
5. Restore Path Priority Order
During an incident, follow this order to protect the uptime priority:
If Core is down but app1-bu is healthy → failover (Section 3), not restore. This is the fastest path back to service.
If a non-Core server is down and S3 is reachable → Standard Restore (see Per-Server Runbooks).
If S3 is unreachable → Emergency Restore using the secondary backup destination defined in the Backup & Security Hardening Standard, before falling back to Hudu-documented manual rebuild.
If both backup destinations and the standby are unavailable → declare a major incident and engage the full DR tabletop response, not a routine runbook.
6. Document Control
This is the live, authoritative DR plan. Any older document describing different failover timing (e.g., 5-minute polling) is superseded as of this version and should be marked:
SUPERSEDED: Replaced by Disaster Recovery Master Plan v2.0 (July 9, 2026). Do not use for active DR procedures.
Companion documents — Backup & Security Hardening Standard, Per-Server Runbooks, and the DR Testing & Validation Schedule — are equally authoritative for their respective scope and are maintained alongside this plan.
Next: see the DR Testing & Validation Schedule for the cadence that converts these targets from committed to verified.
references/2-Backup-Security-Standard.md
+137
View File
@@ -0,0 +1,137 @@
IT PRO PARTNER — DOCUMENT 2 OF 4
Backup & Security Hardening Standard
Immutability, credential separation, and a second backup destination
Field
Value
Document status
ACTIVE
Date
July 9, 2026
Applies to
All servers listed in the Infrastructure Inventory
Owner
Network Services Team
1. Why This Exists
The original backup standard put system config, Docker volumes, database dumps, and Hermes state all in one Wasabi S3-compatible bucket, written using the same credentials the production servers use for everything else. That means a compromised or misbehaving production server could delete or overwrite its own backups — and it means a single provider/account outage takes down backup storage, restore source, and warm-standby sync simultaneously. This document closes both gaps.
2. Backup Immutability — Required, Not Planned
2.1 Object Lock
Enable Object Lock (compliance mode) on the primary Wasabi bucket for all daily and 15-minute snapshot objects. Once written, an object cannot be deleted or overwritten until its retention period expires — including by an account holder with full credentials. This is the single highest-value control against ransomware and accidental deletion.
Data Class
Object Lock Retention
Hermes state (15-min snapshots)
72 hours
Daily system/config/DB/app backups
90 days
Weekly rollups (new — see Section 5)
6 months
Monthly rollups (new — see Section 5)
24 months
2.2 Bucket versioning
Enable versioning in addition to Object Lock, so an overwrite creates a new version rather than destroying the prior one, even for objects outside the Object Lock retention window.
2.3 Credential separation — write-only vs. restore/admin
Production servers must only ever hold write-only credentials. They can create new backup objects; they cannot list, read, delete, or modify existing ones. A separate, tightly held restore/admin credential — used only by a human or the DR automation during an actual restore — is the only credential with read/delete rights.
{
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": "arn:aws:s3:::hermes-vps-backups/<server>/*"
}
// Explicitly no s3:GetObject, s3:DeleteObject, s3:ListBucket on the production-server credential
Production server IAM/API keys: PutObject only, scoped to that server's own prefix.
Restore/admin credential: full read + write + delete, held outside production servers (password manager / vault), MFA-gated where the provider supports it.
No server should hold a credential capable of deleting its own historical backups.
2.4 Access logging and alerting
Enable bucket access logging on the Wasabi bucket.
Alert on any DeleteObject, mass overwrite, or use of the restore/admin credential outside a declared incident window.
Alert if a production server's write-only credential is used for anything other than PutObject.
3. Second Backup Destination — Required
Wasabi remains the primary backup and restore path because it is already integrated. A second, independently-owned destination is added purely as insurance against a Wasabi account, bucket, or provider-level failure — it is not a replacement for the primary path.
Layer
Provider
Purpose
Sync Method
Primary
Wasabi (existing bucket)
Normal daily backup + restore path
Direct write from each server (current process)
Secondary
Independent provider/account (e.g. Backblaze B2, or a second Wasabi account in a different region)
Protects against primary provider/account failure or compromise
One-way replication job, primary → secondary, nightly
Local standby cache
app1-bu disk
Fast failover without depending on any S3 endpoint at the moment of failover
Continuous sync of Core/Hermes state only
Why a separate account, not just a separate bucket: If the Wasabi account itself is compromised or suspended, a second bucket in the same account offers no protection. The secondary destination must sit under different credentials and ideally a different provider entirely.
Replication approach: a scheduled job (e.g., rclone) reads from the primary bucket using the restore/admin credential and writes to the secondary destination. Production servers never write directly to the secondary — this keeps the write-only credential model intact and avoids doubling the attack surface.
4. Local Standby Cache (app1-bu)
The warm standby must not depend on reaching S3 at the exact moment of failover. app1-bu keeps a locally cached copy of the most recent Hermes state, refreshed continuously (targeting the same 15-minute RPO as the S3 snapshots), so it can start service even if the S3 endpoint is temporarily unreachable during the outage that triggered failover in the first place.
5. Retention Model
Flat 90-day daily / 48-hour snapshot retention is fine for routine recovery but offers no protection against slow-developing corruption discovered after 90 days. Add tiered rollups:
Backup Type
Retention
15-minute Hermes snapshots
72 hours
Daily backups
90 days
Weekly rollups
6–12 months
Monthly rollups
12–36 months
Yearly archive
Optional — compliance-dependent
6. Per-Application Backup Inventory — Verification Required
The backup standard states Docker volumes and database dumps are covered daily. Confirm this is true for every host, not assumed. Track it in a living table:
Host
Docker Volumes Enumerated?
Included in Backup Job?
DB Dump Scheduled?
Last Verified
Core
☐
☐
☐
app1
☐
☐
☐
app2
☐
☐
☐
app3
☐
☐
☐
app1-bu
☐
☐
N/A
wphost02
☐
☐
☐ (MariaDB)
fleettracker360
☐
☐
☐ (MariaDB)
Known gap to close first: wphost02 and fleettracker360 both run MariaDB — confirm scheduled dumps exist, upload to S3, are non-empty, and are restorable. Add an alert: no successful DB backup for a service in more than 26 hours = critical.
7. Monitoring & Alerting Requirements
Backup job success/failure — alert same day on failure.
Backup age / RPO compliance — alert if the newest backup for any service exceeds its RPO target.
Standby sync freshness — alert if app1-bu's local cache falls more than 15 minutes stale.
Standby cannot reach S3 — alert immediately, since this affects both backup and failover readiness.
Disk usage on every host — warning at 80%, critical at 90%, emergency at 95%.
Mass delete / unusual access on the backup bucket.
Next: see Per-Server Runbooks for the exact restore commands per host, and the DR Testing & Validation Schedule to verify these controls are actually working.
references/3-Per-Server-Runbooks.md
+227
View File
@@ -0,0 +1,227 @@
IT PRO PARTNER — DOCUMENT 3 OF 4
Per-Server Runbooks
Exact restore steps, dependencies, and validation checks for every host
Field
Value
Document status
ACTIVE — fill in bracketed [ ] fields with live DNS/port/secret values
Date
July 9, 2026
Companion documents
DR Master Plan v2; Backup & Security Hardening Standard; DR Testing & Validation Schedule
Each runbook below follows the same structure so responders can find what they need under pressure without hunting. Restore commands use the standard base-install and S3 pull process from the Server Provisioning Standard. Fields in [brackets] are site-specific values to confirm and fill in — they are structural placeholders, not unknowns to guess at.
Core
Hermes orchestration hub — highest priority, protected by warm standby (app1-bu)
Field
Value
Tier
Light — Hetzner/netcup RS 2000 G12, 8C/16GB/512GB, Debian 13
RTO / RPO target
5 min / 15 min (see DR Master Plan v2, Section 2)
Dependencies
app1-bu (standby), S3 backup bucket, [DNS provider], [external monitor service]
DNS records
[list A/CNAME records pointing to Core, e.g. hermes.itpp.internal]
Firewall ports
22 (SSH, key-only), [Hermes gateway port], [any inter-service ports]
Backup path
s3://hermes-vps-backups/live/ (15-min), s3://hermes-vps-backups/core/ (daily)
Required secrets
[.env for Hermes gateway], [DB credentials if applicable], itpp-infra SSH key
Restore Procedure
If app1-bu is healthy, use failover (DR Master Plan v2, Section 3) instead of a full restore — it is faster and is the primary recovery path for Core.
If a full rebuild is required: provision a Light-tier server, run base install (hostname, updates, timezone, fail2ban, node-exporter, ufw).
Deploy itpp-infra SSH key; disable password auth and root login.
Pull latest Hermes state: aws s3 cp s3://hermes-vps-backups/live/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
Pull latest daily config/backup set from s3://hermes-vps-backups/core/latest/.
Restore Docker stacks: cd /opt/<service> && docker compose up -d
Do not accept live traffic until health checks below pass.
Health & Validation Checks
Hermes gateway responds on its health endpoint
MEMORY.md loads without truncation or parse errors
Last Hermes state timestamp is within 15 minutes of restore time
active-lock.json reflects Core as active_node before releasing standby
Failback
Follow the full failback sequence in DR Master Plan v2, Section 3.3 — do not improvise during an incident.
Known Failure Modes
Core restored from a stale backup while app1-bu holds newer state — mitigated by the mandatory sync-before-start step in failback
Both Core and app1-bu active simultaneously — mitigated by the active-lock file and fencing
MEMORY.md over-pruned or corrupted on restore — cross-check against the priority-tagging scheme before resuming automation
app1-bu (Warm Standby)
Standby tier, activates the Hermes gateway when Core is confirmed down
Field
Value
Tier
Standby — Hetzner CPX11, 2C/2G/40GB, Debian 12
Role
Passive by default; becomes active only via the failover sequence in DR Master Plan v2, Section 3
Dependencies
Local state cache, S3 (secondary path), external monitor for quorum check
DNS records
[failover target / low-TTL record used to redirect traffic on activation]
Firewall ports
22 (SSH, key-only), [Hermes gateway port, opened only on activation]
Backup path
N/A — consumes Core's backups; does not independently back up production data
Required secrets
Same Hermes gateway .env as Core, kept in sync but inactive until failover
Restore Procedure
app1-bu itself is restored like any Standby-tier host if it fails: provision replacement CPX11, base install, deploy itpp-infra key.
Re-establish the local state cache sync from Core (Backup & Security Hardening Standard, Section 4).
Confirm standby readiness checklist below before trusting it as a failover target again.
Health & Validation Checks
Can reach primary and secondary S3 backup destinations
Local state cache is less than 15 minutes stale
Hermes gateway package/image present and startable
Required secrets present and match Core's current .env
Disk usage under 80%
active-lock.json reachable and readable
Known Failure Modes
Standby capacity marginal under real load — see DR Master Plan v2, Section 4 load-test action item
Local cache silently falls stale without alerting — mitigated by the sync-freshness alert in the Backup & Security Hardening Standard
app1 / app2 / app3
Standard tier application hosts
Field
Value
Tier
Standard — RS 4000 G12, 12C/32GB/1TB NVMe, Debian 13
RTO / RPO target
60 min / 24 hours
Dependencies
[list per-app: DB, reverse proxy, upstream APIs]
DNS records
[A/CNAME records per app]
Firewall ports
22 (SSH), 80/443 (Caddy, if serving web)
Backup path
s3://hermes-vps-backups/<server>/
Required secrets
/opt/<service>/.env (mode 600) per stack
Restore Procedure
Provision a Standard-tier server (same tier or better).
Run base install (Section 2 of DR Master Plan / provisioning standard).
Deploy itpp-infra SSH key; disable password auth and root login.
aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
cd /opt/<service> && docker compose up -d for each stack.
Point DNS/traffic at the restored host only after health checks pass.
Docker Volume Restore
docker volume ls — confirm all expected volumes are present after restore.
docker inspect <volume> — confirm mount paths match the pre-incident layout.
Spot-check restored volume contents against the last known-good file or record.
Health & Validation Checks
App-specific health endpoint returns 200
Docker Compose stack reports all containers healthy
Restored data timestamp is within the 24-hour RPO window
Known Failure Modes
Docker volume not actually included in the backup job — verify against the inventory table in the Backup & Security Hardening Standard before assuming restore is complete
wphost02
WordPress host — MariaDB-backed
Field
Value
Tier
[confirm tier — Standard or Legacy]
RTO / RPO target
2 hours / 24 hours
Dependencies
MariaDB, [WordPress plugins/uploads directory], Caddy
DNS records
[WordPress site domain(s)]
Firewall ports
22, 80/443
Backup path
s3://hermes-vps-backups/wphost02/
Required secrets
WordPress DB credentials, [any API keys in wp-config.php]
Restore Procedure
Provision replacement host matching current tier.
Run base install; deploy itpp-infra SSH key.
aws s3 cp s3://hermes-vps-backups/wphost02/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
Restore Docker/Caddy stack: cd /opt/wordpress && docker compose up -d
Database Restore
Confirm the latest MariaDB dump exists and is non-empty in the restored backup set.
Import the dump into a fresh MariaDB instance before pointing WordPress at it.
Verify wp-config.php DB credentials match the restored instance.
Spot-check a known page/post to confirm data integrity.
Health & Validation Checks
Site loads over HTTPS with valid cert
WordPress admin login succeeds
Latest DB dump timestamp is within 24 hours (RPO compliance)
Known Failure Modes
Scheduled DB dump not actually running — flagged as a known gap; verify before trusting this runbook's RPO claim
Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
fleettracker360
Fleet tracking application — MariaDB-backed
Field
Value
Tier
[confirm tier — Standard or Legacy]
RTO / RPO target
2 hours / 24 hours
Dependencies
MariaDB, [application-specific services]
DNS records
[fleettracker360 domain(s)]
Firewall ports
22, [app port(s)]
Backup path
s3://hermes-vps-backups/fleettracker360/
Required secrets
DB credentials, [any third-party API keys]
Restore Procedure
Provision replacement host matching current tier.
Run base install; deploy itpp-infra SSH key.
aws s3 cp s3://hermes-vps-backups/fleettracker360/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
Restore Docker stack: cd /opt/fleettracker360 && docker compose up -d
Database Restore
Confirm latest MariaDB dump exists and is non-empty.
Import into a fresh instance before pointing the app at it.
Validate against a known vehicle/record ID.
Health & Validation Checks
Application health endpoint returns 200
Latest DB dump timestamp is within 24 hours
Known Failure Modes
Scheduled DB dump not actually running — same known gap as wphost02; verify before trusting this runbook's RPO claim
Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
Legacy Hetzner Hosts
CPX21 / CPX41 boxes not yet migrated to the current standard
Field
Value
Tier
Legacy — CPX21 or CPX41, Debian 12
RTO / RPO target
4 hours / 24 hours
Dependencies
[list per host]
DNS records
[list per host]
Firewall ports
[list per host]
Backup path
s3://hermes-vps-backups/<server>/
Required secrets
[per host]
Restore Procedure
If S3 is reachable: provision replacement, base install, restore from S3 per the Standard Restore process.
If S3 is unreachable: use Hetzner rescue mode to reinstall OS, redeploy itpp-infra key, and rebuild from Hudu-documented configs (Emergency Restore).
Plan to migrate to Debian 13 and current tier standards on next rebuild rather than restoring like-for-like.
Health & Validation Checks
Service-specific health check for whatever this host runs
Confirm restored config matches the most recent Hudu documentation
Known Failure Modes
Emergency Restore path depends entirely on Hudu documentation being complete and current — audit this quarterly
Fill in all [bracketed] fields before this document is relied upon in a live incident. Track completion in the DR Testing & Validation Schedule.
references/4-DR-Testing-Schedule.md
+144
View File
@@ -0,0 +1,144 @@
IT PRO PARTNER — DOCUMENT 4 OF 4
DR Testing & Validation Schedule
The cadence that turns committed RTO/RPO targets into verified ones
Field
Value
Document status
ACTIVE
Date
July 9, 2026
Governing rule
A target in the Master Plan is “unverified” until it has passed a live test recorded here.
1. Test Cadence
Test
Frequency
Owner
Backup job success check
Daily (automated)
Monitoring/alerting
Backup age / RPO compliance check
Daily (automated)
Monitoring/alerting
Single database restore
Monthly
Network Services Team
Single app stack restore
Quarterly
Network Services Team
Core failover test (Core → app1-bu)
Quarterly
Network Services Team
Core failback test (app1-bu → Core)
Quarterly, immediately after failover test
Network Services Team
Full DR tabletop exercise
Twice yearly
Full team + stakeholders
Provider/S3 outage simulation
Twice yearly
Network Services Team
Do not skip the failback test: A failover test without an immediate failback test only proves half the design. Schedule them back-to-back so failback is never left untested for a full quarter.
2. Restore Test Log — Required Fields
Every restore test, whether scheduled or incident-driven, gets logged with:
Date and tester name
Server/application tested
Backup artifact path and timestamp used
Restore duration (actual, compared against the RTO target)
Data validation performed against a known record, file, or document
Issues encountered
Remediation ticket, if needed
Date
Tester
System
Backup Used
Duration
RTO Target Met?
Validation Passed?
Issues / Ticket
3. Failover / Failback Test Log
Date
Trigger Simulated
Detection Time
Time to Active
Failback Time
Split-Brain Check Passed?
Issues / Ticket
4. Immediate Priority Test Queue
Given uptime is the top priority, run these before anything else on the cadence above:
Standby capacity load test on app1-bu (validates DR Master Plan v2, Section 4).
First Core failover + failback test, end to end, to establish a real baseline against the 5-minute RTO target.
Verify MariaDB dump automation on wphost02 and fleettracker360, then restore one dump from each.
Restore one Docker volume on each of app1, app2, app3 into a test environment.
Confirm Object Lock and write-only IAM are actually enabled on the Wasabi bucket (Backup & Security Hardening Standard, Section 2), not just documented.
Stand up and validate the secondary backup destination replication job.
5. Sign-off
A target moves from “committed” to “verified” in the Master Plan only after it has a passing entry in this log, dated within the current quarter. Re-verify quarterly — infrastructure and application changes can silently break a previously-passing test.
Target
Status
Last Verified
Verified By
Core RTO: 5 minutes
Unverified — pending first test
Core RPO: 15 minutes
Unverified — pending first test
app1/2/3 RTO: 60 minutes
Unverified — pending first test
wphost02 / fleettracker360 RTO: 2 hours
Unverified — pending first test
Backup immutability (Object Lock)
Unverified — pending confirmation
Secondary backup destination
Not yet built
This document is the source of truth for whether the DR plan actually works, not just whether it is well written. Keep it current.
references/admin-ai-migration-plan.md
+216
View File
@@ -0,0 +1,216 @@
# Admin-AI Migration Plan: Hetzner → app1
**Source:** 178.156.167.181 (Hetzner CPX41, 93% disk)
**Target:** 152.53.36.131 (app1, RS 4000, 3% disk)
**DNS:** admin-ai.itpropartner.com
## Current State
### Source (Old AI Box)
- LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
- 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
- Open WebUI (0.9.6) — 121 MB SQLite DB
- Ollama — 8 local models (~10 GB), 7 cloud-proxy models
- Qdrant vector DB — small, optional
- Caddy reverse proxy
- 226G disk at 93% — critical threshold
### Target (app1)
- Docker + Compose v5.3.1 already installed
- Ollama + Open WebUI + n8n already running
- Caddy 2.6.2 active on 80/443
- 1TB at 3% used, 32 GB RAM
- Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com
## Migration Plan
### Phase 0 — Pre-flight (do now, before you leave for Colombia)
No downtime. No DNS changes. Prep work only.
- [ ] Pull LiteLLM image on app1
- [ ] Set up LiteLLM + Postgres docker-compose on app1
- [ ] Export Postgres DB from old → import to app1
- [ ] Add Caddy config for admin-ai.itpropartner.com on app1
- [ ] Test admin-ai endpoint on app1 via direct IP or internal DNS
- [ ] Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b)
### Phase 1 — Cutover (when you're ready)
- [ ] Switch Hermes to fallback chain (OpenRouter deepseek-chat)
- [ ] Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
- [ ] Monitor admin-ai endpoint
- [ ] Run end-to-end model test through app1
- [ ] Verify vision (gemini-pro-latest) works
### Phase 2 — Restore Primary
- [ ] Switch Hermes back to admin-ai primary
- [ ] Verify all 22 cron jobs pass
- [ ] Verify Anita's profile
### Phase 3 — Decommission (after 3 days stable)
- [ ] Keep old box running for rollback
- [ ] After verification, decom Hetzner AI server
## Detailed Steps
### Step 0: Pull LiteLLM image on app1
```bash
ssh -i /root/.ssh/itpp-infra root@152.53.36.131
docker pull ghcr.io/berriai/litellm:v1.84.0
```
### Step 1: Create LiteLLM docker-compose on app1
Create `/root/docker/litellm/docker-compose.yml` and `/root/docker/litellm/.env`:
```yaml
services:
litellm:
image: ghcr.io/berriai/litellm:v1.84.0
container_name: litellm
restart: always
environment:
- DATABASE_URL=${DATABASE_URL}
- STORE_MODEL_IN_DB=True
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
- LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
- GEMINI_API_KEY=${GEMINI_API_KEY:-}
- PERPLEXITYAI_API_KEY=${PERPLEXITYAI_API_KEY:-}
volumes:
- ./config.yaml:/app/config.yaml
command: ["--config", "/app/config.yaml", "--port", "4000"]
ports:
- "127.0.0.1:4000:4000"
networks:
- litellm-net
depends_on:
litellm_postgres:
condition: service_healthy
litellm_postgres:
image: postgres:16
container_name: litellm_postgres
restart: always
environment:
- POSTGRES_DB=litellm_db
- POSTGRES_USER=litellm
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
volumes:
- litellm_db_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm -d litellm_db"]
interval: 5s
timeout: 5s
retries: 5
networks:
- litellm-net
networks:
litellm-net:
driver: bridge
volumes:
litellm_db_data:
```
`.env` file:
```
POSTGRES_PASSWORD=litellm
DATABASE_URL=postgresql://litellm:litellm@litellm_postgres:5432/litellm_db
LITELLM_MASTER_KEY=sk-lit...c5b1
LITELLM_SALT_KEY=sk-lZ0..._v3A
```
`config.yaml`:
```yaml
general_settings:
master_key: "sk-lit...c5b1"
use_dashboard: true
database_url: postgresql://litellm:***@litellm_postgres:5432/litellm_db
store_model_in_db: true
database_connection_pool_limit: 10
database_connection_timeout: 30
enforce_database: true
```
### Step 2: Export/import Postgres DB
On old server:
```bash
PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql
```
Copy to Core, then to app1:
```bash
scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/
```
On app1, start Postgres first, then import:
```bash
cd /root/docker/litellm && docker compose up -d litellm_postgres
sleep 5
PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql
```
### Step 3: Start LiteLLM on app1
```bash
cd /root/docker/litellm && docker compose up -d litellm
```
### Step 4: Test on app1 via loopback
```bash
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lit...c5b1"
```
### Step 5: Add Caddy config on app1
Add to /etc/caddy/Caddyfile:
```
admin-ai.itpropartner.com {
reverse_proxy 127.0.0.1:4000
}
```
### Step 6: DNS cutover
```bash
# Set TTL 60 first, wait, then update
hermes config set model.fallbacks '[...openrouter fallback...]'
# Flip DNS A record to 152.53.36.131
# Verify: curl https://admin-ai.itpropartner.com/v1/models
```
## Rollback
If cutover fails:
1. DNS back to 178.156.167.181
2. Insert master key hash into old DB:
```bash
HASH=$(echo -n "sk-lit...c5b1" | sha256sum | cut -d' ' -f1)
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
-c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
VALUES ('$HASH', 'sk-...c5b1', 'default_user_id', 0, '{}') \
ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...c5b1';"
```
3. Switch Hermes back to admin-ai
## Provider API Keys (needed for new server)
The Postgres dump carries encrypted API keys (LITELLM_SALT_KEY dependent). As long as the same SALT_KEY is used on app1, the keys migrate transparently.
Direct provider keys in old .env:
- OPENAI_API_KEY: (empty — uses OpenRouter)
- ANTHROPIC_API_KEY: (empty — uses OpenRouter)
- GEMINI_API_KEY: (empty — uses OpenRouter)
- PERPLEXITYAI_API_KEY: (empty — not used)
All model API keys are stored in Postgres LiteLLM_ProxyModelTable, encrypted with SALT_KEY.
references/application-inventory.md
+56
View File
@@ -0,0 +1,56 @@
# Infrastructure Application Inventory
This document provides a comprehensive inventory of all services, applications, and databases running across the infrastructure.
## Servers & Services
### Primary Host
| Application | How It Runs | Ports | Backup Method | Dependencies | Status |
|---|---|---|---|---|---|
| Hermes Agent | systemd | Telegram gateway | Full backup + live sync | admin-ai (LiteLLM), Ollama | LIVE |
| Ollama | systemd | 11434 | none yet | GPU/RAM | LIVE |
| Vaultwarden | Docker | web | none yet | none | LIVE |
| DocuSeal | Docker | web | none yet | Postgres | LIVE |
| SearXNG | Docker | web | none yet | none | LIVE |
| Ops Portal | Python/uvicorn | 8090 | none yet | SyncroMSP API | LIVE |
| **OSINT API** | **Python/FastAPI** | **8100** | **Source at /opt/osint-api/** | **Redis, holehe, maigret, twilio** | **LIVE** |
| Shark Game | Python/uvicorn | web | none yet | SQLite | LIVE |
### Host A (app1 / AI Box)
| Application | How It Runs | Ports | Backup Method | Dependencies | Status |
|---|---|---|---|---|---|
| LiteLLM | Docker | proxy | none yet | Postgres, API keys | MIGRATING |
| Open WebUI | Docker | web | none yet | LiteLLM | MIGRATING |
| Ollama | systemd | 11434 | none yet | GPU/RAM | LIVE |
| n8n | Docker | web | none yet | Postgres | PLANNED |
### Host B (app2)
| Application | How It Runs | Ports | Backup Method | Dependencies | Status |
|---|---|---|---|---|---|
| UNMS/UISP | Docker | 80/443/2055 | auto-backup | Postgres, RabbitMQ, SiriDB | MIGRATING |
| UniFi Controller | Native/Docker | 8443 | auto-backup (.unf) | MongoDB | MIGRATING |
| Traccar | Docker | 8082 | none yet | MySQL/Postgres | PLANNED |
### Host C (app3) / Legacy Web Host (wphost02)
| Application | How It Runs | Ports | Backup Method | Dependencies | Status |
|---|---|---|---|---|---|
| WordPress sites (15+) | RunCloud/Docker | 80/443 | MainWP+WPvivid→S3 | MariaDB, PHP-FPM, Nginx | MIGRATING |
### Docker Box
| Application | How It Runs | Ports | Backup Method | Dependencies | Status |
|---|---|---|---|---|---|
| Uptime Kuma | Docker | 3001 | none yet | SQLite | LIVE |
| nginx-proxy-manager | Docker | 80/81/443 | none yet | SQLite | LIVE |
| RustDesk | Docker | 21114-21119 | none yet | none | LIVE |
## Databases Summary
| Database Engine | Host/Location | Usage / Associated Apps | Details / Backup Status |
|---|---|---|---|
| MariaDB | wphost02 (migrating to Host C) | WordPress sites | 8 databases (Largest: MainWP 137MB) |
| Postgres | Host A (app1) | LiteLLM, n8n (Planned) | none yet |
| Postgres | Host B (app2) | UNMS/UISP, Traccar (Planned) | auto-backup via UNMS |
| Postgres | Primary Host | DocuSeal | none yet |
| MongoDB | Host B (app2) | UniFi Controller | auto-backup via UniFi (.unf) |
| SQLite | Primary Host | Shark Game | none yet |
| SQLite | Docker Box | Uptime Kuma, nginx-proxy-manager | none yet |
references/architecture-audit-2026-07-13.md
+559
View File
@@ -0,0 +1,559 @@
# Server Architecture Plan — Independent Audit
**Audited:** July 13, 2026 00:21 EDT
**Auditor:** Sho'Nuff (Hermes Agent)
**Purpose:** Review approved architecture, identify gaps, present to unbiased 3rd‑party reviewer
**Scope:** Full architecture ...
Legend for all following sections:
| Symbol | Meaning |
|---|---:|
| ✅ | Sound / no issue found |
| ⚠️ | Concern — needs discussion or mitigation |
| ❌ | Flaw — contradicts stated goal or creates risk |
| — | Not applicable / excluded by directive |
---
## 1. Guiding Principles (from User Memory / Context)
| Principle | Source |
|---|---|
| Fault isolation matters — no all-in-one servers | Memory |
| Reliability / performance > savings | Memory |
| No single points of failure | Memory |
| Offline recovery capability required | Memory |
| Provider diversity for DR | Memory |
| Uptime / restorability > cost / convenience | Memory |
| Model / API spend now a real constraint | Memory |
| Clean direct solutions over proxy hops | Memory |
| Do only what is asked; no scope creep | Memory |
| Everything not needed for Hermes runtime should move off Core | This request |
| Ollama excluded from plan entirely | This request |
| Plan must survive independent 3rd‑party review | This request |
---
## 2. Current Live State (observed Jul 13 00:21 EDT)
### 2.1 Core (152.53.192.33 — RS 2000 G12)
| Resource | Value |
|---|---:|
| vCPU | 8 (non-dedicated) |
| RAM | 15 GB |
| Disk | 503 GB (16% used, ~406 GB free) |
| Docker | 11 containers running |
| systemd services | 33 running |
**Docker containers on Core:**
| Container | Purpose | Needed for Hermes to function? |
|---|---|---|
| vaultwarden | Password vault | No |
| komodo-core-1 + komodo-mongo-1 | Docker management | No |
| prometheus | Metrics | No |
| grafana | Dashboards | No |
| twenty-server-1 + twenty-worker-1 + twenty-db-1 + twenty-redis-1 | CRM (Twenty) | No |
| docuseal | Document signing | No |
| searxng | Meta-search engine | No (Super Search uses it but runs as separate service) |
**systemd services on Core relevant to architecture:**
| Service | Purpose | Needed for Hermes to function? |
|---|---|---|
| hermes.service | Hermes Agent gateway | **Yes** |
| hermes-browser.service | Headless Chromium for browser tool | **Yes** |
| hermes-assistant.service | PWA backend | **Yes** |
| ollama.service | Local LLM server | **No** (to be removed per this directive) |
| caddy.service | Reverse proxy | Partially — only for core.itpropartner.com paths; full Caddy can move |
| ops-portal.service | Ops Portal FastAPI backend | No |
| osint-api.service | OSINT API | No |
| osint-person.service | OSINT Person MCP | **Yes** (runs locally, needed for Hermes) |
| super-search.service | Super Search MCP | **Yes** (runs locally, needed for Hermes) |
| dre-mcp.service | DRE MCP | **Yes** (runs locally, needed for Hermes) |
| twilio-mcp.service | Twilio MCP | **Yes** (runs locally, needed for Hermes) |
| mysql-tunnel.service | MySQL SSH tunnel to Apex DB | **Yes** (needed for mysql MCP tool) |
| shark-game.service | Shark Game backend | No |
| node_exporter.service | Prometheus exporter | No |
| strongswan-starter.service | VPN | No (for remote access; not Hermes-critical) |
| xl2tpd.service | L2TP VPN | No |
**Current Hermes provider/failover chain:**
```text
Primary: gpt-5.5 via admin-ai (https://admin-ai.itpropartner.com/v1)
Fallback 1: deepseek-v4-pro via admin-ai
Fallback 2: deepseek/deepseek-chat via OpenRouter (https://openrouter.ai/api/v1)
Fallback 3: llama3.2:3b via ollama-local (http://localhost:11434/v1)
```
Delegation (subagent) primary: `deepseek-v4-pro` via admin-ai.
admin-ai.itpropartner.com currently points to: **178.156.167.181 (Hetzner old-ai box).**
### 2.2 app1 (152.53.36.131 — RS 4000 G12)
| Resource | Value |
|---|---:|
| vCPU | 12 dedicated EPYC 9645 |
| RAM | 32 GB |
| Disk | 1 TB (3% used, ~943 GB free) |
Running: Ollama, Open WebUI, n8n + n8n-postgres, Caddy.
### 2.3 app2 (152.53.39.202 — RS 4000 G12)
Running: UNMS/UISP stack, UniFi Controller (deployed Jul 12).
### 2.4 app3 (152.53.241.111 — RS 4000 G12)
Target for WordPress/Apex from SiteGround/wphost02.
### 2.5 core-bu (5.161.114.8 — Hetzner CPX11, was app1-bu)
Warm standby. Hermes gateway dormant. Watchdog checks Core every 5 min. Needs CPX11→CPX31 upgrade for DR capacity.
### 2.6 old-ai (178.156.167.181 — Hetzner CPX41)
Current admin-ai LiteLLM host. 93% disk. Target for decommission.
### 2.7 Remaining Hetzner hosts
| Host | IP | Status |
|---|---|---|
| docker | 178.156.168.35 | To decommission |
| old app1 | 87.99.144.163 | Deprecated |
| unms | 5.161.225.131 | DNS to app2 pending |
| unifi | 143.198.185.17 | DNS updated Jul 12, decom pending |
| hudu | 178.156.130.130 | Ready to migrate |
| fleettracker | 178.156.149.32 | Traccar ready to move |
| wphost02 | 5.161.62.38 | WP to app3 |
| tony-vps | 87.99.159.142 | Keep — Tony's Hermes |
---
## 3. Ollama Audit — Why It Should Be Excluded
Per directive: **"Exclude/strike Ollama from the plan altogether."**
### 3.1 What Ollama provides today
| Function | Model | Reality |
|---|---|---|
| Local LLM fallback | `llama3.2:3b` (2 GB, loaded on demand) | Extremely limited; useful only for "I am alive" pings. Not competent for any real work. |
| Attempted fallback model | `llama3.3:70b` (42 GB disk) | **Cannot load.** Requires ~35-40 GB RAM. Core has 15 GB. This model is wasted disk space and should be deleted immediately regardless of architecture decision. |
### 3.2 Problems with Ollama on Core
| ❌ | Issue |
|---|---|
| ❌ | 42 GB of wasted disk (`llama3.3:70b`) that can never be used |
| ❌ | Creates false sense of security — "we have local fallback" — but the usable model is too dumb for any actual work |
| ❌ | `llama3.2:3b` cannot handle structured JSON, tool calling, code fixes, or even basic email summarization reliably |
| ❌ | RAM pressure: Ollama service consumes RAM even when idle, competing with Hermes gateway + browser + Postgres (future LiteLLM DB) |
| ❌ | Maintenance overhead: model updates, service health checks, logging |
| ❌ | Slows Core boot time |
### 3.3 Verdict
| Verdict |
|---|
| ❌ Ollama should be completely removed from Core. Delete `llama3.3:70b` immediately. Remove `ollama.service`. Remove from failover chain. |
| ❌ `llama3.2:3b` as last-resort fallback is an architectural lie — it creates the impression of resilience while providing no practical value. If all remote providers fail, Hermes should report "all providers exhausted" rather than silently degrading to a model that cannot perform any task. |
---
## 4. Replacement Models for Low-Lift Tasks
Per directive: **"Present other model options for low-lift tasks that Ollama could handle."**
### 4.1 What "low-lift" means in practice
Ollama was supposed to handle:
- Survival pings (acknowledge a message exists)
- Very basic classification (is this email spam?)
- Simple summarization of short text
- "I'm alive" health checks
None of these require a local model. They can be handled by dirt-cheap remote APIs with far better results.
### 4.2 Cheapest available APIs (Jul 2026)
From live pricing research:
| Model | Provider | Input/1M | Output/1M | Context | Notes |
|---|---:|---:|---:|---:|---|
| **GPT-4.1 Nano** | OpenAI/OpenRouter | $0.10 | $0.40 | 1M | Cheapest competent model; good enough for classification/light summarization |
| **Gemini 2.0 Flash / Flash-Lite** | Google/OpenRouter | $0.10 | $0.40 | 1M | Free tier available; multimodal |
| **DeepSeek V4 Flash** | DeepSeek direct | $0.14 | $0.28 | 1M | Already in admin-ai; cheaper than Ollama electricity |
| **Mistral Small** | Mistral/OpenRouter | $0.10 | $0.30 | 128K | Good for classification |
| **GLM-4.7-Flash** | Z.ai | **Free** | **Free** | 128K | Actually free tier; limited but usable |
| **GLM-4.6V-Flash** | Z.ai | **Free** | **Free** | — | Free vision model |
| **Qwen Qwen3-Flash** | Alibaba/OpenRouter | $0.07 | $0.14 | 32K | Ultra cheap |
| **Llama 3.1 8B** | Various/OpenRouter | $0.05–$0.10 | $0.05–$0.20 | 128K | Open model, many providers; dirt cheap |
### 4.3 Recommended replacement tier
| Tier | Model | Route | Cost estimate | Use case |
|---|---:|---|---|---|
| **Survival ping** | `deepseek-v4-flash` | admin-ai | ~$0.14/M input | Already has access; extremely cheap; 1M context; way better than llama3.2:3b |
| **Cheapest possible** | `openrouter/qwen/qwen3-flash` | OpenRouter | ~$0.07/$0.14 | Classification, trivial summarization |
| **Free option** | `openrouter/z-ai/glm-4.7-flash` | OpenRouter | Free | Low-risk cron jobs, health checks, zero-cost classification |
| **When admin-ai is down** | `openrouter/deepseek/deepseek-chat` | OpenRouter | ~$0.14/$0.28 | Already in fallback chain; keep |
| **When OpenRouter is also down** | Report "all providers exhausted" | — | — | Honest failure > silent useless degradation |
### 4.4 Cost comparison: Ollama vs cheapest API
| Scenario | Ollama cost | API cost |
|---|---|---|
| Per month electricity/RAM | ~$5–$10 (portion of VPS cost for RAM + CPU cycles) | ~$0.01–$0.50 for all survival-ping volume |
| Quality | Unusable for real work | Usable for light tasks |
| Maintenance | Updates, service management, disk cleanup | Zero |
| Disk | 2–42 GB wasted | Zero |
**Verdict:** The cheapest API is economically better AND qualitatively better than Ollama for every single use case that Ollama was supposed to cover.
### 4.5 The honest fallback design
Remove the survival-model pretense:
```text
Fallback 1: deepseek-v4-pro via admin-ai (different model, same provider)
Fallback 2: deepseek/deepseek-chat via OpenRouter (different provider)
Fallback 3: deepseek-v4-flash via admin-ai (cheapest competent fallback)
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter (dirt cheap, still better than Ollama)
Fallback 5: openrouter/z-ai/glm-4.7-flash via OpenRouter (free tier — zero cost last resort)
```
Or, simpler:
```text
Fallback 1: deepseek-v4-pro via admin-ai
Fallback 2: deepseek-v4-flash via admin-ai
Fallback 3: deepseek/deepseek-chat via OpenRouter
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter
```
When all are exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter status." No silent degradation.
---
## 5. Architecture Audit — Original Plan vs Reality
### 5.1 The approved rebalance plan
From `server-architecture-plan` (rebalanced Jul 12):
```text
Core: Hermes + LiteLLM + Ollama fallback
app1: Caddy (all domains), Ops Portal, OSINT API, Super Search MCP,
SearXNG, Grafana, cron, n8n + Postgres, Docker services
app2: UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target)
app3: WordPress / web apps
core-bu: Warm standby
```
### 5.2 Problems found
#### 5.2.1 ❌ admin-ai.itpropartner.com DNS cutover risk
**Plan says:** LiteLLM → Core. DNS stays (implied localhost or unchanged).
**Reality:** Today admin-ai.itpropartner.com resolves to 178.156.167.181 (Hetzner old-ai). The `core-rebalance-plan.md` says "admin-ai.itpropartner.com → Core IP (stays)" — but this is contradictory. The domain currently points to old-ai, not Core. Moving LiteLLM to Core means DNS must change, OR Hermes must switch from `admin-ai.itpropartner.com` to `localhost`. The plan does not address this.
**Impact:** If LiteLLM moves to Core and Hermes uses `https://admin-ai.itpropartner.com/v1`, then during the DNS propagation window:
- Hermes on Core tries to reach admin-ai.itpropartner.com
- DNS may still point to old-ai
- Old-ai LiteLLM may already be down
- Result: Hermes falls back to OpenRouter unnecessarily
**Fix:** Either:
- (a) Change Hermes `providers.admin-ai.base_url` to `http://localhost:4000/v1` after LiteLLM is running locally (no DNS dependency), OR
- (b) Switch admin-ai.itpropartner.com DNS to Core IP as a deliberate step in the migration, accepting the propagation window.
Option (a) is cleaner and eliminates DNS as a failure mode for Hermes→admin-ai entirely. The domain `admin-ai.itpropartner.com` would still serve external clients.
#### 5.2.2 ⚠️ core-bu fails over to OpenRouter-only
**Plan says:** core-bu takes over when Core is down.
**Problem:** If Core hosts LiteLLM, then a Core outage kills both Hermes AND admin-ai simultaneously. core-bu's Hermes would have zero access to admin-ai. It would fall to OpenRouter.
**Is this acceptable?** Yes — IF we accept the tradeoff. But the plan does not acknowledge this explicitly.
**Mitigation:** core-bu should be pre-configured to use OpenRouter as primary during failover. The current standby config already uses admin-ai, which would be dead. This needs an explicit config change for core-bu's Hermes.
#### 5.2.3 ⚠️ RAM on Core with LiteLLM + Postgres
| Component | Estimated RAM |
|---|---:|
| Hermes gateway + browser + assistant | ~2-3 GB |
| LiteLLM + Postgres (Docker) | ~1-2 GB |
| OSINT Person + Super Search + DRE + Twilio MCPs | ~1 GB |
| Caddy (if kept) | ~100 MB |
| MySQL tunnel | ~200 MB |
| System + buffers | ~2 GB |
| Docker daemon | ~500 MB |
| Core Docker services to be evacuated | ~3-4 GB (freed) |
| **Total after evacuation** | **~7-9 GB of 15 GB** |
This is **acceptable but tight**. LiteLLM under load can spike Postgres memory. No headroom for Ollama (which we're removing anyway). No headroom for spikes.
**Recommendation:** Document the RAM budget explicitly. Monitor Postgres memory after migration. Be prepared to add LiteLLM resource limits.
#### 5.2.4 ❌ `core-rebalance-plan.md` is too sparse
The full rebalance plan is only 25 lines. It lacks:
- Rollback steps per phase
- Verification gates between phases
- DNS propagation wait times
- Service dependency ordering
- What to do if a phase fails
- Cron job re-pinning steps
- Anita profile verification step
- Vision model verification step
- Gateway model override check
- S3 sync impact during migration
This is not a reviewable plan. It is an outline.
#### 5.2.5 ⚠️ Caddy on Core is both needed and not needed
Core needs Caddy for:
- `core.itpropartner.com` paths (Vaultwarden, Twenty, DocuSeal, etc.) — but these are moving to app1
- Hermes PWA / browser tool — internal-only? or public?
If all web services move to app1, Core's Caddy becomes only:
- Hermes PWA backend proxy
- Possible browser tool endpoint
**Question:** After rebalance, does Core need a public Caddy at all? If Hermes is internally accessed and all web services are on app1, Core could run with no public ports except SSH and VPN.
#### 5.2.6 ⚠️ MCP services running on Core but serving app1-hosted tools
Super Search depends on SearXNG (Docker container). After rebalance, SearXNG moves to app1. Super Search's server.py references `http://127.0.0.1:8888/search` for SearXNG.
**Impact:** Super Search on Core would be calling app1's SearXNG across the network. This introduces latency and a cross-server dependency where none existed before.
**Fix:** Either:
- (a) Point Super Search's SearXNG URL at app1's internal/private IP, OR
- (b) Keep a minimal SearXNG on Core for Super Search, OR
- (c) Move Super Search MCP to app1 alongside SearXNG
Option (c) is cleanest but then Super Search is not local to Hermes — it would be accessed via MCP over HTTP between servers.
#### 5.2.7 ❌ Ollama is in the plan
Both `server-architecture-plan` and `core-rebalance-plan` reference Ollama as a fallback. Per the directive, this must be removed.
#### 5.2.8 ❌ The plan fails the "fault isolation" principle
Core becomes:
```text
Hermes + LiteLLM + all MCPs + Postgres + Caddy + Tailscale + WireGuard + SSH
```
This is a single box that:
- Runs the agent
- Runs the model proxy
- Runs all MCP tool servers
- Runs the reverse proxy
- Runs the VPN
A Core outage is now a **total outage** — no agent, no model access, no tools, no remote access. Even with core-bu failover, core-bu would have no admin-ai and limited MCP access.
Compare to the alternative:
```text
Core: Hermes + MCPs (tools only)
app1: LiteLLM + Caddy + all web services + Super Search + SearXNG + Grafana + cron
Core outage → core-bu takes over Hermes, admin-ai still up on app1
app1 outage → Core Hermes loses admin-ai, falls to OpenRouter; web services down but agent still alive
```
This alternative provides partial degradation. Both boxes must fail simultaneously for a total outage.
---
## 6. Revised Architecture Recommendation
### 6.1 Target roles (Ollama removed)
| Box | Model | Role | Critical for Hermes? |
|---:|---|---|---|
| **Core** | RS 2000 | **Hermes Agent + essential MCPs only** | Yes |
| **app1** | RS 4000 | **LiteLLM (admin-ai) + Caddy (all domains) + Ops Portal + Super Search + SearXNG + OSINT API + Grafana + cron + n8n + Postgres + all Docker services** | Partially — admin-ai is critical; web services are not |
| **app2** | RS 4000 | UNMS/UISP, UniFi, Hudu, Traccar | No |
| **app3** | RS 4000 | WordPress, Apex, web apps | No |
| **core-bu** | CPX31 | Warm standby for Core | Yes (DR) |
### 6.2 Core — keep only what Hermes needs to function
| Keep on Core | Reason |
|---|---|
| hermes.service | Core function |
| hermes-browser.service | Browser tool for Hermes |
| hermes-assistant.service | PWA backend |
| osint-person.service | OSINT Person MCP (local = fast, no cross-server latency) |
| super-search.service | Super Search MCP (local; SearXNG endpoint updated to app1) |
| dre-mcp.service | DRE MCP |
| twilio-mcp.service | Twilio MCP |
| mysql-tunnel.service | MySQL SSH tunnel to Apex DB |
| caddy.service | Only if needed for Hermes PWA; otherwise remove |
| tailscaled.service | Private infrastructure access |
| ssh.service | Administrative access |
| cron.service | System cron |
| docker.service | Only if MCPs containerized; otherwise remove |
| Move off Core → target | Service |
|---|---|
| → app1 | vaultwarden (Docker) |
| → app1 | komodo-core + komodo-mongo (Docker) |
| → app1 | prometheus + grafana (Docker) |
| → app1 | twenty-server + worker + db + redis (Docker) |
| → app1 | docuseal (Docker) |
| → app1 | searxng (Docker) |
| → app1 | ops-portal.service (systemd) |
| → app1 | osint-api.service (systemd) |
| → app1 | shark-game.service (systemd) |
| → app1 | node_exporter.service (systemd) |
| → app1 | caddy.service (systemd — if Core doesn't need it) |
| → Delete | ollama.service + ollama models |
| → Delete | strongswan-starter.service + xl2tpd.service (VPN — replace with Tailscale-only or keep if needed) |
### 6.3 Revised fallback chain (no Ollama)
```text
Primary: gpt-5.5 via admin-ai (app1)
Fallback 1: deepseek-v4-pro via admin-ai (app1)
Fallback 2: deepseek-v4-flash via admin-ai (app1)
Fallback 3: deepseek/deepseek-chat via OpenRouter
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter
```
When all exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter."
### 6.4 Revised DNS plan
| Domain | Current → | Target |
|---|---|---|
| admin-ai.itpropartner.com | 178.156.167.181 (old-ai) | **152.53.36.131 (app1)** |
| ops.itpropartner.com | Core | **app1** |
| core.itpropartner.com | Core | **app1** (all web services move) |
| sign.itpropartner.com | Core | **app1** |
| n8n.itpropartner.com | Already app1 | No change |
| ai.itpropartner.com | Already app1? | No change |
Hermes on Core uses:
```text
providers.admin-ai.base_url = https://admin-ai.itpropartner.com/v1
```
This means admin-ai DNS must point to app1 before Hermes can use it. The DNS cutover for admin-ai is **the critical path**. All other DNS changes can happen independently.
### 6.5 Revised migration phases
#### Phase 0 — Pre-flight (no downtime)
1. Inventory old-ai admin-ai completely
2. Pull LiteLLM image on app1
3. Create LiteLLM docker-compose on app1
4. Dump old-ai Postgres → import to app1
5. Start app1 LiteLLM, test locally
6. Verify all models work through app1 local LiteLLM
7. Verify encrypted provider keys survived (SALT_KEY match)
8. Move non-Hermes Docker services: Core → app1
9. Remove Ollama from Core (stop service, delete models, disable service)
#### Phase 1 — DNS cutover (planned window)
1. Switch Hermes to OpenRouter-only fallback temporarily
2. SiteGround DNS: admin-ai.itpropartner.com → 152.53.36.131
3. Wait for DNS propagation (check from Core: `dig +short admin-ai.itpropartner.com`)
4. Caddy on app1 issues cert for admin-ai.itpropartner.com
5. Verify: `curl https://admin-ai.itpropartner.com/v1/models` from Core
6. Restore Hermes admin-ai primary + full fallback chain
7. Test full model chain: gpt-5.5, deepseek-v4-pro, deepseek-v4-flash
#### Phase 2 — Remaining DNS and service moves
1. Caddy → app1
2. ops.itpropartner.com → app1
3. core.itpropartner.com → app1
4. sign.itpropartner.com → app1
5. Move systemd services (ops-portal, osint-api, shark-game) → app1
6. Move cron jobs → app1 (or re-create on app1)
7. Update Super Search SearXNG URL to app1's SearXNG
#### Phase 3 — Verification
1. Verify Hermes works with new admin-ai (app1)
2. Verify core-bu Hermes standby config
3. Verify all cron jobs
4. Verify Anita profile
5. Verify vision model
6. Verify gateway sessions have no stale model overrides
#### Phase 4 — Stabilization (7 days)
- Keep old-ai running for rollback
- Compare LiteLLM spend logs old vs new
- Monitor Core RAM/CPU after evacuation
- Confirm Caddy cert auto-renewal on app1
- Confirm all MCP tools function
#### Phase 5 — Decommission
- old-ai (178.156.167.181) → cancel
- docker (178.156.168.35) → cancel
- remaining Hetzner hosts → cancel as each service confirms migration
#### Rollback per phase
| Phase | Rollback action |
|---|---|
| Phase 1 | DNS back to old-ai IP; restore Hermes to admin-ai. Insert master key hash into old DB if needed. |
| Phase 2 | DNS back to Core IPs; restart services on Core. |
| Phase 3 | Fix individual failures without full rollback. |
| Phase 4 | If app1 admin-ai unstable, Phase 1 rollback. |
---
## 7. Open Questions for 3rd-Party Review
These are areas where the plan has ambiguity that a reviewer should flag:
1. **Does Core need a public Caddy after all web services move?** If Hermes PWA and browser tool are internal-only, Core could run with SSH + Tailscale only. This simplifies Core dramatically.
2. **Should admin-ai be on Core or app1?** This audit recommends app1 for fault isolation. The user's earlier intuition (Core colocation) favors simplicity. Both arguments are presented above. The reviewer should weigh them.
3. **Is core-bu's CPX11 sufficient for DR if admin-ai moves to app1?** CPX11 has 2 vCPU / 2 GB RAM. Hermes gateway alone may fit, but subagent delegation and browser tools may struggle. The plan notes CPX31 upgrade is needed.
4. **What happens to Super Search if SearXNG moves to app1?** Cross-server dependency is introduced. Should Super Search move to app1 too? Or should a minimal SearXNG stay on Core?
5. **Are the free-tier models (GLM-4.7-Flash) reliable enough for the last fallback tier?** Free tiers can disappear without notice. The plan should include a fallback that doesn't depend on a free tier.
6. **Should VPN (WireGuard/StrongSwan) stay on Core or move?** If Core goes down and VPN is on Core, remote access to the entire infrastructure is lost. VPN on a separate box (or redundant) would be safer.
---
## 8. Summary of Changes from Original Plan
| Original Plan Element | Audit Finding | Revised |
|---|---|---|
| Ollama on Core | Removed entirely | Deleted. Replaced with cheap API fallbacks |
| admin-ai → Core | Risk of single-point failure | admin-ai → app1 |
| Core keeps Caddy + all Docker services | Breaches "Core is lightweight" principle | Move all non-Hermes services to app1 |
| 3-step migration outline | Too sparse; no rollback | 5-phase plan with per-phase rollback |
| DNS plan vague | admin-ai DNS path unclear | Explicit DNS cutover with propagation check |
| Fallback chain ends at Ollama | Useless fallback | Chain ends at cheap-but-competent API, not local model |
| No core-bu config plan | Standby would use dead admin-ai | core-bu must use OpenRouter during Core outage |
| Super Search ↔ SearXNG co-location broken | Cross-server dependency introduced | Explicitly flagged; recommendation to move Super Search to app1 |
| 3-day stabilization | Too short | 7-day stabilization |
---
**End of audit. Ready for 3rd-party review.**
references/backup-policy.md
+72
View File
@@ -0,0 +1,72 @@
# IT Pro Partner - Backup and Disaster Recovery Policy
**Effective Date:** July 10, 2026
**Document Owner:** IT Operations Manager / MSP Backup Administrator
**Scope:** This policy applies to all systems, data, and configurations managed by IT Pro Partner, including internal operations and client environments.
## 1. Overview
This policy defines the backup strategies, retention schedules, recovery objectives, and testing procedures for all managed environments. Our backup infrastructure relies primarily on off-site cloud storage via Wasabi S3 to ensure data durability, availability, and separation of concerns.
## 2. Classification and Recovery Point Objectives (RPO)
Data is classified into three tiers to align with business continuity requirements:
| Classification | Description | Target RPO | Examples |
| :--- | :--- | :--- | :--- |
| **Critical** | Highly dynamic state and session data essential for real-time operations. | 10 - 15 Minutes | AI state databases, memory consolidation, active session configs. |
| **Important** | Daily production workloads, client websites, and core infrastructure configs. | 24 Hours | WordPress sites, Hermes full instances, router configurations, project documentation (`/root/projects/`). |
| **Standard** | Static environments, base server images, or ad-hoc archival workloads. | 7 Days / N/A | Hetzner server snapshots, one-time manual node backups (e.g., wphost02). |
## 3. Backup Schedule and Inventory
The following systems are actively protected under automated backup routines:
| Asset / System | Backup Type / Tool | Schedule | Storage Destination |
| :--- | :--- | :--- | :--- |
| **Hermes Live Sync** | Incremental (state.db, sessions, configs) via `hermes-live-sync.sh` | Every 15 minutes | Wasabi S3 |
| **Memory Consolidation** | Pre-prune state via `backup-before-prune` | Every 10 minutes | Wasabi S3 |
| **Hermes Full Backup** | Full System Archive via `hermes-backup.sh` | Daily at 01:00 UTC | Wasabi S3 |
| **Project Documentation** | Included in root essentials backup via `root-essentials-backup.sh` | Daily | Wasabi S3 |
| **Router Configs** | Configuration dumps via `wisp-backup.py` | Daily | Wasabi S3 (`mikrotik-ccr-backups`) |
| **WordPress Sites (15)**| Incremental & Full via MainWP + WPvivid Pro | Daily, Weekly, Monthly | Wasabi S3 (`ipp-mainwp-s3`) |
| **Hetzner Servers** | Full VM Snapshots via `snapshot-hetzner.py`| Weekly (Mondays 05:00 UTC) | Hetzner Cloud Native Storage |
| **wphost02 Node** | One-time full server backup | Ad-hoc / Manual (Jul 10) | Wasabi S3 |
## 4. Retention Policy
Unless specifically overridden by a distinct client contract, all backups adhere to the following baseline retention periods:
- **Daily Backups:** Retained for 7 days.
- **Weekly Backups:** Retained for 30 days.
- **Monthly Backups:** Retained for 12 months.
- **Continuous / Sub-hourly Syncs (Critical Tier):** Retained for 48 hours before rolling up into daily aggregates.
## 5. Storage and Security
### 5.1 Storage Locations
All primary off-site backups are stored in Wasabi S3 object storage to leverage immutability (where configured) and high durability. Specific buckets are isolated by workload (e.g., `ipp-mainwp-s3` for client sites, `mikrotik-ccr-backups` for infrastructure).
### 5.2 Encryption Standards
- **At Rest:** Wasabi server-side encryption (SSE) is enabled by default across all backup buckets. MainWP/WPvivid WordPress backups are encrypted prior to upload.
- **In Transit:** All backup transfers occur exclusively over TLS 1.2+ secure connections.
- **Client-Side:** Native client-side encryption is not universally implemented on custom shell scripts at this time (see Section 8).
## 6. Verification and Testing
Consistent validation is critical to ensure data integrity and reliable disaster recovery.
- **Automated Verification:** A daily audit watchdog script monitors the success/failure state of scheduled cron jobs, alerting the operations team of missed or failed jobs.
- **Restore Testing:** A comprehensive, documented restore test of a randomly selected asset (WordPress site, Hermes state, or Router config) is conducted **Monthly**.
## 7. Roles and Responsibilities
- **Backup Owner:** The IT Operations Manager is accountable for the overarching backup strategy, budget, and policy adherence.
- **Backup Administrator:** Responsible for day-to-day monitoring, script maintenance, provisioning new buckets, and executing the monthly restore tests.
## 8. Acknowledged Gaps and Roadmap
The following known limitations exist within the current architecture and are targeted for future remediation:
1. **Client-Side Encryption:** Some shell-script backups (e.g., Hermes full backups) rely entirely on server-side S3 encryption and TLS. Client-side GPG/AES encryption prior to transit is planned.
2. **Docker Volumes:** Automated granular backups of individual Docker persistent volumes are not fully standardized across all infrastructure nodes.
3. **Database Dumps:** Outside of WordPress/MainWP workflows, bare-metal MySQL/PostgreSQL database dumps are not yet uniformly automated.
references/core-rebalance-plan.md
+1436
View File
File diff suppressed because it is too large Load Diff
references/docker-management-research.md
+483
View File
@@ -0,0 +1,483 @@
# Docker Management — Multi-Host Tool Research
**Date:** 2026-07-09
**Context:** IT Pro Partner runs Docker Compose stacks across multiple servers (Core/netcup, ai.itpropartner.com, docker, unms, etc.). We need a single tool to manage all containers across hosts. Mixed environment of Docker Compose stacks (no Swarm/Kubernetes).
---
## Evaluation Criteria (Priority Order)
1. **Recovery manual completeness** — can Germaine use it without Sho'Nuff?
2. **Multi-host support** — can it see/manage containers on different servers?
3. **CLI/API access** — can I script against it from Hermes?
4. **Self-hosted** — runs on our infrastructure, not SaaS
5. **Free/open source** — no subscription fees for personal use
6. **Web GUI** — nice bonus, not required
7. **Webhook support** — can trigger actions from external events
---
## Tool Comparison Table
| Feature | Portainer | Arcane | Komodo | Dockge | Dockhand | Lazydocker |
|---|---|---|---|---|---|---|
| **Self-hosted** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ (local TUI) |
| **Free tier** | 3 nodes BE free | ✅ 100% FOSS | ✅ 100% FOSS | ✅ 100% FOSS | ✅ Homelab free | ✅ FOSS |
| **Multi-host** | ✅ Agent-based | ✅ Agent-based | ✅ Agent-based | ✅ Agents (v1.4+) | ✅ Agent/TLS | ❌ Single host |
| **REST API** | ✅ Full API | ✅ OpenAPI 3.1 | ✅ REST + WebSocket | ❌ No API | ⏳ Roadmap (distant) | ❌ |
| **Web GUI (1-5)** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ❌ (TUI) |
| **Setup complexity** | Easy | Easy | Medium | Easy | Easy | Very Easy |
| **License** | Proprietary (BE) / MIT (CE) | BSD-3-Clause | GPL-3.0 | MIT | BSL 1.1 → Apache 2.0 (2029) | MIT |
| **Stars** | ~30k | 6.3k | ~3k | 23.7k | 5.1k | 38.7k |
| **Webhooks** | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ❌ No |
| **Compose focus** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Compose-only | ✅ Yes | ❌ Container focus |
| **Swarm support** | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ❌ Not mentioned | ❌ No |
| **K8s support** | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
---
## 1. Portainer
**Website:** https://www.portainer.io/
**Docs:** https://docs.portainer.io/
**GitHub:** https://github.com/portainer/portainer
### Overview
Industry-standard Docker management UI. Mature, battle-tested, vast ecosystem. Used by teams of all sizes.
### Pricing
- **Business Edition (BE) — 3 nodes free forever** (full features, no time limit, requires annual license key renewal but $0)
- **Home & Student** — $155/yr for up to 15 nodes (non-commercial only)
- **Business Starter** — $105/mo for 5 nodes (commercial)
- **Business Scale** — $209/mo for 5 nodes
- **Community Edition (CE)** — Free forever but missing RBAC, GitOps, templates, and other BE features
### Multi-host Support
✅ **Yes** — via **Portainer Agent** (lightweight Go daemon on each remote host). Agent connects to Portainer Server. Also supports Docker API over TCP/TLS directly.
### REST API
✅ **Excellent** — comprehensive REST API with Swagger/OpenAPI docs. All UI actions available via API. Official Go client library.
### Setup Complexity: **Easy**
```bash
docker run -d -p 9001:9001 --name portainer_agent --restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
portainer/agent:latest
```
One command per remote host, add environment in Portainer UI.
### Recovery Documentation: **Excellent**
- Extensive official docs at https://docs.portainer.io/
- Backup/restore guides: https://docs.portainer.io/advanced/backup
- Large community, thousands of tutorials
- **Germaine-friendly:** Yes — the UI is intuitive, and the docs are thorough
### Pros
- Most mature, most documented
- Free 3-node BE tier covers our needs
- Comprehensive REST API for Hermes scripting
- Agent auto-updates, mTLS support
- Webhooks for GitOps/auto-deploy
### Cons
- Beyond 3 nodes requires payment
- Can feel heavy/bloated for simple Compose management
- CE version lacks features (use BE free tier instead)
- Proprietary license (BE requires license key renewal)
---
## 2. Arcane ⭐ (Strong Contender)
**Website:** https://getarcane.app/
**Docs:** https://getarcane.app/docs/setup/installation
**GitHub:** https://github.com/getarcaneapp/arcane
**Live Demo:** https://demo.getarcane.app/
### Overview
Modern Docker management UI built with SvelteKit (frontend) + Go (backend). BSD-3-Clause licensed. Clean, fast, well-designed. Gaining traction rapidly (6.3k stars, 70 contributors).
### Pricing
- **100% free and open source** — BSD-3-Clause license
- No paid tiers, no feature gating
- Mobile app (iOS TestFlight beta) — also free
### Multi-host Support
✅ **Yes** — via **Arcane Agent** on remote hosts. Two connection modes:
- **Direct** — Manager connects to Agent on TCP 3553 (requires inbound port)
- **Edge** — Agent dials out to Manager (for NAT/firewall, no inbound port needed)
- Transport modes: gRPC tunnel or periodic polling (poll mode for idle environments)
### REST API
✅ **Excellent** — "Fully Documented API" built with Huma on Gin, featuring auto-generated OpenAPI 3.1 documentation. Go client library available.
### Setup Complexity: **Easy**
```bash
# Manager (one line)
docker run -d --name arcane -p 3552:3552 -v arcane_data:/data ghcr.io/getarcaneapp/manager:latest
# Agent on each remote host
docker run -d --name arcane-agent -p 3553:3553 \
-v /var/run/docker.sock:/var/run/docker.sock \
-e AGENT_MODE=true \
-e AGENT_TOKEN=<token> \
-e MANAGER_API_URL=http://manager:3552 \
ghcr.io/getarcaneapp/agent:latest
```
### Recovery Documentation: **Good**
- Documentation website at getarcane.app
- Active Discord community
- Growing collection of guides and tutorials
- **Germaine-friendly:** Yes — modern, intuitive UI, good docs
### Key Features
- Container, image, volume, network management
- Docker Compose stack management
- Docker Swarm support
- Image vulnerability scanning (built-in)
- Image building from Dockerfiles
- Resource monitoring (CPU, memory, network)
- RBAC and OIDC/SSO
- GitOps lifecycle hooks
- iOS mobile app
- Webhook triggers
### Pros
- 100% free, no paid tiers ever (BSD-3-Clause)
- Modern, fast, beautiful UI (SvelteKit)
- Excellent REST API with OpenAPI 3.1
- Edge mode for NAT/firewall'd hosts
- Active development (3k+ commits, frequent releases)
- Growing community
### Cons
- Newer project (less battle-tested than Portainer)
- Smaller community (but growing fast)
- No Kubernetes support (not needed for our use case)
- No LDAP/AD (OIDC/SSO available)
---
## 3. Komodo ⭐ (Strong Contender)
**Website:** https://komo.do/
**Docs:** https://komo.do/docs/intro
**GitHub:** https://github.com/moghtech/komodo
**Blog Guide:** https://blog.saltdata.ro/managing-docker-komodo
### Overview
API-first build and deployment system written in Rust (backend) + TypeScript (frontend). GPL-3.0 licensed. Core + Periphery agent architecture. Includes CI/CD pipeline capabilities alongside Docker management.
### Pricing
- **100% free and open source** — GPL-3.0
- No paid tiers, no feature gating
- Unlimited servers — "There is no limit to the number of servers you can connect, and there never will be."
### Multi-host Support
✅ **Yes** — via **Periphery** agent on each remote host. Periphery communicates with Core over a bi-directional WebSocket connection. Can run as systemd service, Docker container, or standalone binary.
### REST API
✅ **Excellent** — REST + WebSocket API with multiple client libraries:
- Komodo CLI (built-in)
- Rust crate (`komodo_client`)
- NPM package (`komodo_client`)
- curl examples in docs
### Setup Complexity: **Medium**
- Core requires MongoDB (or FerretDB/Postgres)
- Docker Compose file with MongoDB + Core
- Periphery agent install via Python script or Docker
- More moving parts than Portainer/Arcane
### Recovery Documentation: **Good**
- Well-organized docs at komo.do
- Built-in backup/restore CLI
- **Germaine-friendly:** Mostly — UI is clean, but setup is more involved
### Key Features
- Docker container and Compose stack management
- Docker Swarm management
- Image building from Dockerfiles (supports AWS EC2 spot instances for build capacity)
- Git repository management on remote servers
- Resource Syncs (GitOps) — auto-deploy on git push
- Procedures — multi-step automation workflows with scheduling
- Server resource monitoring (CPU, RAM, disk)
- RBAC with granular permissions
- OAuth/OIDC authentication
- Full audit trail of every change
- Webhook triggers
### Pros
- Feature-rich — more than just Docker management (builds, CI/CD, automation)
- Excellent API with first-class client libraries (ideal for Hermes scripting)
- Truly unlimited servers
- GPL-3.0 open source
- Active development
- Backup/restore built-in
### Cons
- Heavier setup (needs MongoDB or FerretDB/Postgres)
- More complex than needed if we only need Docker management (CI/CD features may go unused)
- Smaller community than Portainer
- Rust backend — harder to extend/modify than Go
---
## 4. Dockge
**Website:** https://github.com/louislam/dockge
**README:** https://github.com/louislam/dockge/blob/master/README.md
### Overview
Focused Docker Compose stack manager by the creator of Uptime Kuma. MIT licensed. Laser-focused on compose.yaml management rather than general Docker management. Very popular (23.7k stars).
### Pricing
- **100% free and open source** — MIT license
### Multi-host Support
✅ **Yes** — since v1.4.0, supports multiple agents. Can manage stacks from different Docker hosts in one interface.
### REST API
❌ **No documented REST API** — UI-focused tool. No API for external scripting.
### Setup Complexity: **Easy**
```yaml
services:
dockge:
image: louislam/dockge:latest
ports:
- 5001:5001
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./data:/app/data
- /opt/stacks:/opt/stacks
environment:
- DOCKGE_STACKS_DIR=/opt/stacks
```
### Recovery Documentation: **Minimal**
- README only, no dedicated docs site
- Basic FAQ in README
- **Germaine-friendly:** Partially — UI is simple but recovery docs are thin
### Key Features
- Create/Edit/Start/Stop/Restart compose.yaml stacks
- Interactive compose.yaml editor
- Interactive web terminal for containers
- Multi-agent support (v1.4+)
- Image update button
- Stack folder scanning
### Pros
- Very simple, focused tool (does one thing well)
- Beautiful, reactive UI
- Lightweight and fast
- Large community (23.7k stars)
### Cons
- **Compose-only** — cannot manage individual containers, networks, volumes outside compose
- **No REST API** — cannot be scripted from Hermes
- No webhook support
- No RBAC/authentication (single-user only)
- No monitoring or alerting
- README-only documentation
- "Can I manage existing stacks?" — requires manual file moves
---
## 5. Dockhand
**Website:** https://dockhand.pro/
**Manual:** https://dockhand.pro/manual/
**GitHub:** https://github.com/Finsys/dockhand
### Overview
Modern Docker management platform with polished Svelte UI. BSL 1.1 licensed (converts to Apache 2.0 in 2029). The code IS source-available but not truly open-source in the free-software sense. 5.1k stars.
### Pricing
- **Free tier** — all features for homelab use (no RBAC, no LDAP, no commercial use)
- **SMB** — $499/host/year
- **Enterprise** — $1,499/host/year
### Multi-host Support
✅ **Yes** — three methods:
1. **Local Docker socket** (/var/run/docker.sock)
2. **Remote TCP connections** with TLS
3. **Hawser agent** — lightweight agent for NAT/firewall traversal (similar to Portainer Edge / Arcane Edge)
### REST API
⚠️ **On roadmap ("distant future")** — No working API today. API keys listed as "distant future" on the roadmap. OpenAPI/Swagger also "distant future."
### Setup Complexity: **Easy**
```bash
docker run -d --name dockhand -p 3000:3000 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v dockhand_data:/app/data \
fnsys/dockhand:latest
```
### Recovery Documentation: **Good**
- Full user manual at dockhand.pro/manual
- Well-organized documentation
- **Germaine-friendly:** Yes — polished UI, good manual
### Key Features
- Container operations (start/stop/restart/create)
- Visual Compose editor
- Git repository deployment with webhook auto-sync
- Vulnerability scanning (Grype/Trivy) with safe-pull protection
- Live CPU/memory metrics
- Real-time log streaming
- OIDC/SSO (free)
- MFA (TOTP)
- Hawser agent for NAT traversal
- RBAC and LDAP/AD (Enterprise only)
- Multi-environment switching
### Pros
- Most polished modern UI (5/5)
- 30-second deployment
- Free for homelab use
- Hawser agent handles NAT/firewall well
- Good documentation
### Cons
- **No REST API** — API/API keys are "distant future" on roadmap. Cannot be scripted from Hermes
- BSL 1.1 license — not truly open source (converts to Apache 2.0 in 2029)
- Paid beyond homelab ($499/host/yr for commercial use)
- RBAC and LDAP/AD only in Enterprise tier ($1,499/host/yr)
- Smaller community (5.1k stars)
- Relatively new project (165 commits)
---
## 6. Lazydocker
**Website:** https://lazydocker.com/
**GitHub:** https://github.com/jesseduffield/lazydocker
### Overview
Terminal TUI (Text User Interface) for Docker. MIT licensed. Extremely popular (38.7k stars). Written in Go by the same author as Lazygit. Runs as a local process on a single host.
### Pricing
- **100% free and open source** — MIT license
### Multi-host Support
❌ **No** — runs locally on each host. Cannot manage remote Docker hosts from one interface. Each host needs its own Lazydocker instance (or SSH into each host).
### REST API
❌ **No** — it's a terminal application, not a server.
### Setup Complexity: **Very Easy**
```bash
# Install
brew install lazydocker # macOS
# Or Docker one-liner
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
lazyteam/lazydocker
```
### Recovery Documentation: **Minimal**
- README with basic usage
- Keyboard shortcuts reference
- **Germaine-friendly:** Terminal-only — requires comfort with CLI/TUI
### Pros
- Very lightweight and fast
- Great for quick SSH-and-check sessions
- Keyboard-navigable, efficient for power users
- No server infrastructure needed
### Cons
- **Single host only** — cannot manage our multi-server setup
- **No API** — cannot be scripted from Hermes
- **No web GUI** — terminal only (not Germaine-friendly)
- No persistent state or history
- No user management or RBAC
---
## Recommendation: 🦎 Komodo ⭐
### Best Fit for IT Pro Partner
Given our criteria and environment:
| Criterion | Winner | Rationale |
|---|---|---|
| Recovery completeness | **Portainer / Komodo** | Both have excellent docs. Komodo has built-in backup CLI. |
| Multi-host support | **All (except Lazydocker)** | All support agents. Pick based on other criteria. |
| CLI/API access | **Komodo** | REST + WebSocket + 3 client libraries (CLI, Rust, npm) + curl examples. Best API story. |
| Self-hosted | **All (except Lazydocker)** | All run on our infra. |
| Free/open source | **Arcane / Komodo** | Both are 100% free with permissive licenses. |
| Web GUI | **Dockhand / Portainer** | Both have polished UIs. Portainer is more mature. |
| Webhooks | **Komodo / Portainer / Arcane** | All three support webhooks. |
### Top 3 Contenders (Ranked)
#### 1st: **Komodo** 🏆
- **Why:** Best API story (REST + WS + multiple client libs) makes it ideal for Hermes scripting. Core + Periphery architecture handles our mixed environment. GPL-3.0, truly free. Built-in backup/restore. Webhook triggers for GitOps. Unlimited servers. Server monitoring included.
- **Trade-off:** Heavier setup (needs MongoDB). More features than we need if we only want Docker management.
#### 2nd: **Arcane** 🥈
- **Why:** 100% free BSD-3-Clause. Beautiful modern UI. Great multi-host support (Direct + Edge modes). Solid OpenAPI 3.1 REST API. Lightweight Go backend. Built-in vulnerability scanning. iOS mobile app.
- **Trade-off:** Newer project with smaller community. Less documentation than Portainer.
#### 3rd: **Portainer** 🥉
- **Why:** Most mature, best documented, largest community. Free 3-node BE tier covers our needs. Comprehensive REST API. Agent is rock-solid.
- **Trade-off:** Beyond 3 nodes costs money. Covers our current needs but doesn't scale free. Proprietary license.
### Why not the others?
- **Dockge:** No API, no webhooks, compose-only. Good for one-off single-user compose management but not for our multi-host, scriptable needs.
- **Dockhand:** No API (roadmap item, "distant future"). BSL license. Paid for commercial use. Great UI but cannot be scripted by Hermes.
- **Lazydocker:** Single host only. Terminal-only. Good for quick checks but not a management platform.
### Implementation Suggestion
Start with **Komodo** for the API-first multi-host management. If the MongoDB dependency is a concern, **Arcane** is a strong alternative that's lighter to deploy.
### Quick Deploy Comparison
**Komodo** (on management server):
```yaml
# docker-compose.yml for Komodo Core
services:
mongo:
image: mongo
restart: unless-stopped
volumes:
- ./data/mongo-data:/data/db
core:
image: ghcr.io/moghtech/komodo-core:2
restart: unless-stopped
ports:
- 9120:9120
depends_on:
- mongo
volumes:
- ./backups:/backups
- ./keys:/config/keys
```
**Komodo Periphery** (on each managed host):
```bash
curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py \
| python3 - \
--core-address="https://core.itpropartner.com" \
--connect-as="$(hostname)" \
--onboarding-key="O-..."
```
---
## Links Summary
| Tool | Website | Docs | GitHub |
|---|---|---|---|
| Portainer | https://www.portainer.io/ | https://docs.portainer.io/ | https://github.com/portainer/portainer |
| Arcane | https://getarcane.app/ | https://getarcane.app/docs/setup/installation | https://github.com/getarcaneapp/arcane |
| Komodo | https://komo.do/ | https://komo.do/docs/intro | https://github.com/moghtech/komodo |
| Dockge | https://github.com/louislam/dockge | README only | https://github.com/louislam/dockge |
| Dockhand | https://dockhand.pro/ | https://dockhand.pro/manual/ | https://github.com/Finsys/dockhand |
| Lazydocker | https://lazydocker.com/ | README only | https://github.com/jesseduffield/lazydocker |
references/dr-issue-log.md
+443
View File
@@ -0,0 +1,443 @@
# DR Issue Log
> Permanent record of all disaster recovery audit findings, root causes, fixes, and verification dates.
> Updated: 2026-07-09
---
## Summary
**Fixed** [OK]
- **DR-001** [HIGH] Caddyfile not backed up -> added to backup scripts
- **DR-002** [MED] Systemd backup path wrong -> corrected to /etc/systemd/system/
- **DR-003** [MED] migration-creds.txt not backed up -> added to backup scope
- **DR-004** [MED] dre-temp-passwords.txt 644->600
- **DR-005** [MED] migration-creds.txt 644->600
- **DR-006** [HIGH] Full backup stale (last Jul 5) -> manual backup ran Jul 10 (527MB), system crontab added at 1 AM daily
- **DR-007** [HIGH] home-router backup cron error -> switched to run-wisp-backup.sh
- **DR-008** [HIGH] MikroTik backup stale -> installed deps, fixed IP
- **DR-010** [MED] Failover timing -> 5min/2min from 10min/3.5min
- **DR-011** [HIGH] S3 buckets system-configs & docker-volumes created, versioned, IAM updated
- **DR-012** [MED] firecrawl-usage-check crash on KeyError 'monthly' -> hardened load()
- **DR-013** [LOW] ops collector could not find Hetzner token -> added .hetzner_token fallback
**Resolved** [INFO]
- **DR-009** app1-bu DR plan mismatch -> server stays warm per design
- **DR-014** [INFO] home-router-daily-backup paramiko error -> verified working via live test Jul 10
**Investigating / Blocked** [PENDING]
- **DR-015** [MED] service-health-check / apex-mail-watchdog failing on real remote outages (wphost02, WireGuard)
- **DR-016** [HIGH] root-essentials-backup silently failing — tarballs created locally but not uploaded to S3 since Jul 12
- **DR-017** [HIGH] WISP CCR tower configs not backed up — only one old config (Jul 7) in mikrotik-ccr-backups
- **DR-018** [MED] wphost02 backup singular — only one backup from Jul 10, no recurring schedule
- **DR-019** [MED] SiteGround WordPress backup not implemented — siteground/ prefix empty
- **DR-011b** [HIGH] system-configs & docker-volumes sync buckets still empty — 0 objects, scripts exist but not populating
---
## 2026-07-08 -- Initial Full DR Audit
### DR-001 -- Caddyfile not backed up `[HIGH] [OK] Fixed`
**Problem**
`/etc/caddy/Caddyfile` was not copied by `hermes-backup.sh` or `hermes-live-sync.sh`. If Core server fails, the reverse proxy config would need to be rebuilt from scratch.
**Root Cause**
Backup scripts were written to cover Hermes config and user directories but omitted system-level config files entirely.
**Fix**
Added `/etc/caddy/Caddyfile` to both `hermes-backup.sh` and `hermes-live-sync.sh` with DR FIX comments dated 2026-07-08.
**Verification**
- [OK] `bash -n` syntax check passed on both scripts
- [OK] Subagent confirmed all paths referenced correctly
---
### DR-002 -- Systemd service backup path wrong `[MED] [OK] Fixed`
**Problem**
`hermes-backup.sh` collected systemd services from `~/.config/systemd/user/` instead of `/etc/systemd/system/`. Real service files (`hermes-agent.service`, `shark-game.service`, etc.) were not backed up.
**Root Cause**
Backup script path pointed to user-level systemd directory instead of system-level.
**Fix**
Corrected path to `/etc/systemd/system/*.service` in `hermes-backup.sh`. Embedded restore script also updated.
**Verification**
- [OK] Post-fix script syntax check
- [OK] Subagent confirmed correct paths
---
### DR-003 -- migration-creds.txt not in backup scope `[MED] [OK] Fixed`
**Problem**
`/root/.hermes/migration-creds.txt` existed but wasn't referenced by any backup script.
**Root Cause**
File was added to the system after backup script was written, never included in scope.
**Fix**
Added to `hermes-backup.sh` file list with `chmod 600` restore instruction.
**Verification**
- [OK] Post-fix syntax check
- [OK] File confirmed present and included
---
### DR-004 -- dre-temp-passwords.txt exposed `[MED] [OK] Fixed`
**Problem**
`/root/.hermes/references/dre-temp-passwords.txt` was readable by all users (644) instead of owner-only (600).
**Root Cause**
Script created the file without explicit permission setting.
**Fix**
`chmod 600`
**Verification**
- [OK] `ls -la` confirms `-rw-------`
---
### DR-005 -- migration-creds.txt exposed `[MED] [OK] Fixed`
**Problem**
Same as DR-004 -- `migration-creds.txt` was 644.
**Root Cause**
Written without explicit permission setting.
**Fix**
`chmod 600`
**Verification**
- [OK] `ls -la` confirms `-rw-------`
---
### DR-006 -- Full backup stale `[HIGH] [OK] Fixed 2026-07-10`
**Problem**
`hermes-full-backup.tar.gz` last uploaded to S3 on Jul 5. Daily 5 AM cron had missed 3 days.
**Root Cause**
No cron job scheduled the backup script. `hermes-backup.sh` existed and was correct but was never wired to cron via Hermes or system crontab. The gateway lifecycle guard (#30719) blocked running it as a Hermes cron job.
**Fix**
- Manually ran `hermes-backup.sh` on Jul 10 — produced 527MB tarball, uploaded successfully
- Added system crontab entry: `0 1 * * * /root/.hermes/scripts/hermes-backup.sh 2>&1 | logger -t hermes-full-backup`
- Added audit watchdog: `0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit`
**Verification**
- [OK] Manual backup completed: `hermes-full-backup-2026-07-10.tar.gz` (527,318,371 bytes) in S3
- [OK] Crontab entry confirmed active: `crontab -l`
- [OK] Audit script in place to verify backup completion at +1h
---
### DR-007 -- home-router backup cron error `[HIGH] [OK] Fixed`
**Problem**
Cron job errored at 06:00 -- backup script failed. SSH export on router produced a stuck `.in_progress` file that blocked SCP.
**Root Cause**
Cron was using `home-router-backup.sh` (old WireGuard tunnel script) instead of the proper `run-wisp-backup.sh` pipeline. Stuck export files blocked SCP -> S3 upload failed.
**Fix**
- Changed cron to use `run-wisp-backup.sh`
- Cleaned stuck `.in_progress` files from router
- Installed missing packages: `paramiko v5.0.0`, `xl2tpd`, `strongSwan`
**Verification**
- [OK] Backup ran end-to-end
- [OK] Config uploaded to `s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-08/`
---
### DR-008 -- MikroTik CCR backup stale `[HIGH] [OK] Fixed`
**Problem**
`mikrotik-ccr-backups` S3 bucket had no uploads since Jul 5. Two compounding root causes prevented backups.
**Root Causes**
1. `wisp-backup.py` failed because `paramiko` was not installed
2. Tower IP in `config.yaml` was `192.168.88.1` (LAN) but SSH is restricted to WireGuard tunnel network `10.77.0.0/24`
**Fix**
- Installed `paramiko v5.0.0`
- Updated tower IP to `10.77.0.2` in `wisp-backup/config.yaml`
- Installed missing VPN stack: `xl2tpd`, `strongSwan`
**Verification**
- [OK] Backup ran end-to-end
- [OK] Config uploaded to S3 successfully
---
## 2026-07-08 -- Failover Logic Update
### DR-009 -- app1-bu DR plan mismatch `[INFO] [OK] Resolved`
**Problem**
DR plan doc stated "offline, boots on demand" but server was running (3 days uptime).
**Root Cause**
DR plan documentation was outdated. Actual design is **warm standby** -- always on with Hermes dormant.
**Fix**
Updated DR plan to reflect warm standby design. Server stays running.
**Verification**
- [OK] Docs corrected
- [OK] Server continues as-is
---
### DR-010 -- Failover timing adjustment `[MED] [OK] Fixed`
**Problem**
Failover detection was too slow: 10-minute check intervals with 3.5-minute confirmation window.
**Change**
- **Check interval:** Every 10 min -> **Every 5 min**
- **Confirmation:** 4 x 60s (3.5 min) -> **4 x 30s (2 min)**
- **Max downtime:** ~13.5 min -> **~7 min**
**Rationale**
Faster detection = shorter failover window. If Core doesn't respond within 2 min of constant checking, app1-bu activates.
**Verification**
- [OK] Cron changed to `*/5 * * * *`
- [OK] Watchdog updated: 30s × 4 cycles = 2 min confirmation
- [OK] Verified via SSH on app1-bu
---
## 2026-07-09 -- Session Findings (Ops/Infra pass)
### DR-006 -- Full backup stale (UPDATE: root cause found) `[HIGH] [PENDING] Blocked`
**Problem**
`hermes-full-backup` in Wasabi (`s3://hermes-vps-backups/hermes-full-backup/`) last uploaded
Jul 5. Ops collector reports it `critical` (age > 72h).
**Root Cause (identified 2026-07-09)**
`hermes-backup.sh` exists and is correct (uploads the full tarball to the right path), but
there is NO Hermes cron job that runs it. The 21 jobs in `cron/jobs.json` include
`hermes-live-sync` (every 15m -> `live/` path, healthy) but nothing that runs the daily full
backup. The DR-006 note about a `run-hermes-backup.sh` wrapper was never wired to cron.
**Fix (pending -- requires cron creation, terminal approval-gated this session)**
Create a daily cron job that runs the full backup, e.g.:
```
hermes cron create --name hermes-full-backup --schedule "0 5 * * *" \
--script hermes-backup.sh --no-agent --deliver local
```
Or via cronjob(action='create', name='hermes-full-backup', schedule='0 5 * * *',
script='hermes-backup.sh', no_agent=True, deliver='local'). After first run, verify the
collector flips this bucket to `ok`.
**Status**
- [PENDING] `hermes-backup.sh` verified correct by inspection
- [PENDING] Cron job must be created to schedule it
---
### DR-011 -- S3 buckets missing + sync not scheduled `[HIGH] [OK] Fixed 2026-07-10`
**Problem**
Ops collector reported `itpropartner-system-configs` and `itpropartner-docker-volumes` as `NoSuchBucket`. These are the normalized buckets from the Jul 8 S3 plan.
**Root Cause**
Two compounding issues:
1. The buckets were never created. The Hermes-User IAM key lacked `s3:CreateBucket`, so they had to be created in the Wasabi Console web UI (manual step, never done).
2. The sync scripts (`hermes-system-config-sync.sh`, `hermes-docker-sync.sh`) exist and are correct, but had no cron job scheduling them.
**Fix (completed Jul 10)**
1. Created both buckets via Wasabi API: `itpropartner-system-configs`, `itpropartner-docker-volumes`
2. Enabled versioning on both buckets via `aws s3api put-bucket-versioning`
3. Updated Hermes-User IAM policy to include both bucket ARNs
4. Verified PUT/GET/DELETE operations work on both buckets
5. Sync scripts confirmed present and executable — cron scheduling pending (tracked as DR-011b)
**Verification**
- [OK] Both buckets exist and respond to S3 operations
- [OK] Versioning enabled on both
- [OK] IAM policy updated with bucket ARNs
- [OK] PUT/GET/DELETE tested successfully
- [PENDING] Sync cron jobs to populate buckets (DR-011b)
---
### DR-012 -- firecrawl-usage-check crash `[MED] [OK] Fixed`
**Problem**
`firecrawl-usage-check` cron errored: `KeyError: 'monthly'` in `track-firecrawl.py summary()`.
**Root Cause**
`load()` returned the raw JSON from `firecrawl-usage.json`. An older state file lacked the
`monthly` key, so `data["monthly"].get(...)` raised KeyError. The loader was not schema-safe.
**Fix**
Rewrote `load()` in `/root/.hermes/scripts/track-firecrawl.py` to start from a defaults dict
and merge the file on top, then coerce `monthly`/`calls`/`total_used` to correct types. This
is forward/backward compatible with partial or corrupt state files.
**Verification**
- [OK] `patch` lint (py_compile) passed
- [OK] Current `firecrawl-usage.json` already contains `monthly` key -> next run will pass
---
### DR-013 -- Ops collector could not read Hetzner token `[LOW] [OK] Fixed`
**Problem**
`ops-status.json` showed `hetzner_servers: [{status: error, message: "HETZNER_API_TOKEN not found"}]`,
so the server inventory on the ops portal was empty.
**Root Cause**
`collect_hetzner_servers()` only looked in `os.environ` and `.env`. The Hetzner token on this
box lives in the file `/root/.hermes/scripts/.hetzner_token` (used by `snapshot-hetzner.py`),
which the collector never checked.
**Fix**
Added a fallback in `collect_hetzner_servers()` to read `/root/.hermes/scripts/.hetzner_token`
when env/.env lookups miss. (`ops-data-collector.py`)
**Verification**
- [OK] `patch` lint passed
- [PENDING] Confirm inventory populates on next collector run (needs .hetzner_token present)
---
### DR-014 -- home-router-daily-backup paramiko error `[INFO] [OK] Resolved 2026-07-10`
**Problem**
Job showed `last_status: error` with `ModuleNotFoundError: No module named 'paramiko'`.
**Root Cause**
The error was from the 06:00 Jul 9 run. paramiko had previously been installed against Python 3.11 (leftover `wisp-backup.cpython-311.pyc`), but the system `python3` is now 3.13.
**Resolution**
paramiko IS present for the current interpreter: `/usr/local/lib/python3.13/dist-packages/paramiko/`. The error was a stale artifact from before the py3.13 install was confirmed.
**Live Test Verification (Jul 10)**
- [OK] Ran `wisp-backup.py` manually — completed successfully
- [OK] WireGuard tunnel to 10.77.0.2 confirmed up
- [OK] Config fetched and uploaded to `s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-10/` (3,898 bytes)
- [OK] Logs fetched and uploaded
- [OK] Exit: 1 OK, 0 Failed
- [OK] No paramiko import error — resolved for python3.13
---
### DR-015 -- health-check / apex watchdog failing on remote outages `[MED] [PENDING] External`
**Problem**
`service-health-check` and `apex-mail-watchdog` report errors every cycle.
**Root Cause**
Genuine remote conditions, not script bugs:
- `wphost02` (5.161.62.38) SSH is refused -> apex watchdog SMTP test cannot run.
- WireGuard tunnel to home router (10.77.0.2) is down -> health check + Home-Router-Watchdog fail.
- Portal mockup on port 8081 and MySQL tunnel targets are also unreachable.
Note: the current `apex-mail-watchdog.sh` does LOGIN-only SMTP tests (no test email sent), so
it is NOT the source of the "apex test emails" complaint -- that was an earlier version.
**Fix**
No code change. These clear when wphost02 SSH and the WireGuard tunnel are restored (part of
the broader migration / home-router work). Documented so the errors are understood, not chased
as script bugs.
**Status**
- [PENDING] External dependency -- resolves when remote hosts/tunnel are back online
---
## 2026-07-15 -- Full DR Audit
### DR-016 -- root-essentials-backup silently failing `[HIGH] [PENDING]`
**Problem**
root-essentials-backup ran at 3 AM daily per system crontab but tarballs were NOT being uploaded to S3. S3 only has Jul 10 and Jul 11. Local tarballs accumulated in /tmp (Jul 12-15, 4 files) proving the script runs and creates archives but the upload step fails silently.
**Root Cause**
The script runs past tar creation (local tarballs exist) but the aws s3 cp upload step fails. With set -euo pipefail the script exits on the upload failure, leaving the tarball in /tmp (cleanup never runs). Exact cause of upload failure TBD -- possible S3 permission change or endpoint issue.
Also: the Caddyfile path on line 30 is wrong (`-C /etc systemd/system/caddy/Caddyfile` should be `-C /etc caddy/Caddyfile`), but this is masked by `|| true`.
**Fix**
Pending investigation. Need to run the script manually with verbose output to identify the upload failure.
**Status**
- [PENDING] Investigating
---
### DR-017 -- WISP CCR tower configs not backed up `[HIGH] [PENDING]`
**Problem**
mikrotik-ccr-backups bucket contains home-gateway configs (daily Jul 9-15) but only one WISP CCR config from Jul 7 (`2026-07-07-config.rsc`). No tower router configs since then.
**Root Cause**
wisp-backup.py script references VPN and CCR towers in its docstring, but only the home gateway configs appear in S3. The script may not be reaching the tower CCRs, or the CCR backup path may be misconfigured in config.yaml.
**Fix**
Pending investigation. Need to review wisp-backup/config.yaml and test connectivity to WISP CCR towers.
**Status**
- [PENDING] Investigating
---
### DR-018 -- wphost02 backup not recurring `[MED] [PENDING]`
**Problem**
Only one wphost02 backup exists on S3: wphost02-backup-2026-07-10.tar.gz (656 MB). No recurring backup schedule. MariaDB dumps on wphost02 and fleettracker360 were flagged as known gaps in the Jul 10 DR audit but not yet resolved.
**Root Cause**
Backup was a one-time manual capture during the Jul 10 DR audit. No cron job was created for recurring wphost02 backups.
**Fix**
Pending. Needs a cron job on wphost02 to run regular backups to S3.
**Status**
- [PENDING] Known gap from Jul 10 audit, not yet resolved
---
### DR-019 -- SiteGround WordPress backup not implemented `[MED] [PENDING]`
**Problem**
siteground/ prefix in hermes-vps-backups is completely empty. No SiteGround WordPress site backups exist on S3. MainWP + WPvivid Pro backs up 15 sites to Wasabi independently, but sites not in MainWP have no S3 backup.
**Root Cause**
Fleet-wide SFTP backup timed out at 600 seconds on Jul 10. No alternative was deployed for sites outside MainWP coverage.
**Fix**
Pending. Either batch SFTP backups in groups of 3-5 or extend MainWP coverage to remaining sites.
**Status**
- [PENDING] Known gap, not yet resolved
---
### DR-011b -- system-configs & docker-volumes sync still empty `[HIGH] [PENDING]`
**Problem**
Both itpropartner-system-configs and itpropartner-docker-volumes buckets exist on Wasabi with versioning enabled, but contain 0 objects. The sync scripts (hermes-system-config-sync.sh, hermes-docker-sync.sh) exist and have correct syntax + venv activation, but no Hermes cron jobs are creating them.
**Root Cause**
Cron jobs were never created for these sync scripts. The hermes-live-sync was split into purpose-specific scripts on Jul 8 but the corresponding cron jobs were deferred.
**Fix**
Create two no_agent cron jobs: hermes-system-config-sync (every 15m) and hermes-docker-sync (every 15m).
**Status**
- [PENDING] Deferred since Jul 10
references/dre-credit-reporting.md
+439
View File
@@ -0,0 +1,439 @@
# DRE Credit Bureau Reporting: Legal Analysis
**Prepared:** July 8, 2026
**Subject:** Whether Debt Recovery Experts (DRE) can legally report debtors to credit bureaus under FCRA and Texas law
**Scope:** Covers both consumer debts and B2B/commercial debts, with personal guarantee considerations
---
## Table of Contents
1. [Executive Summary](#1-executive-summary)
2. [FCRA Framework — The Federal Law](#2-fcra-framework--the-federal-law)
3. [Can DRE Report B2B Debts to Consumer Credit Bureaus?](#3-can-dre-report-b2b-debts-to-consumer-credit-bureaus)
4. [Legal Requirements Before Reporting a Consumer Debt](#4-legal-requirements-before-reporting-a-consumer-debt)
5. [Dispute and Investigation Process](#5-dispute-and-investigation-process)
6. [Texas-Specific Laws (Finance Code Chapter 392)](#6-texas-specific-laws-finance-code-chapter-392)
7. [Liability Risks for Incorrect Reporting](#7-liability-risks-for-incorrect-reporting)
8. [Alternative: Business Credit Bureaus](#8-alternative-business-credit-bureaus)
9. [Practical Recommendations for DRE](#9-practical-recommendations-for-dre)
10. [Key Citations](#10-key-citations)
---
## 1. Executive Summary
**Yes — DRE can legally report debtors to consumer credit bureaus (Equifax, Experian, TransUnion) for consumer debts**, provided it complies with FCRA furnisher duties, FDCPA pre-reporting notification requirements, and Texas Finance Code Chapter 392 procedures.
**For B2B (commercial) debts**, the analysis is more nuanced:
- **Corporate debtors (business entities):** The FCRA defines "consumer" as an individual (15 U.S.C. § 1681a(c)). A corporation, LLC, or partnership has no "consumer report" under FCRA. DRE **cannot report a purely corporate B2B debt** to Equifax/Experian/TransUnion consumer credit bureaus — those bureaus do not maintain credit files for business entities, and FCRA does not apply.
- **Individual debtors (sole proprietors, personal guarantors):** If a B2B debt is personally guaranteed by an individual, or if the debtor is a sole proprietor operating under a DBA, DRE **can report that debt against the individual's personal credit file** — but only in compliance with all FCRA and FDCPA requirements, because the individual is a "consumer" under the law.
**For purely B2B debts**, DRE should use **business credit bureaus** (Dun & Bradstreet, Experian Business, Equifax Business) instead of consumer credit bureaus.
---
## 2. FCRA Framework — The Federal Law
### 2.1 Key Definitions (15 U.S.C. § 1681a)
| Term | Definition | Citation |
|------|-----------|---------|
| **Consumer** | An individual (a natural person) | § 1681a(c) |
| **Consumer report** | Communication by a CRA bearing on a consumer's creditworthiness, used for personal, family, or household credit, employment, or other authorized purpose | § 1681a(d)(1) |
| **Consumer reporting agency (CRA)** | Person that regularly assembles consumer credit information for furnishing consumer reports to third parties | § 1681a(f) |
| **Person** | Individual, partnership, corporation, trust, estate, cooperative, association, government entity | § 1681a(b) |
### 2.2 DRE as a Furnisher of Information
Under the FCRA, DRE is a **furnisher of information** — a person who provides information about a consumer to a CRA. The key obligations are in **15 U.S.C. § 1681s-2**:
#### Duties under § 1681s-2(a) — Duty to Provide Accurate Information
- **No reporting with knowledge of errors:** DRE shall not furnish information to a CRA if it knows or has reasonable cause to believe the information is inaccurate (§ 1681s-2(a)(1)(A)).
- **No reporting after consumer notice of error:** Once a consumer notifies DRE in writing that specific information is inaccurate, DRE shall not continue furnishing that information until it confirms accuracy (§ 1681s-2(a)(1)(B)).
- **Duty to correct and update:** DRE must notify all CRAs of any corrections to previously furnished information (§ 1681s-2(a)(2)).
- **Duty to notify of disputed information:** If a consumer directly disputes the debt to DRE, DRE may not report the debt as delinquent during the dispute period without noting the dispute status (§ 1681s-2(a)(3)).
- **Duty to establish reasonable procedures:** DRE must implement reasonable policies to ensure accurate reporting and to respond to notifications of error (§ 1681s-2(a)(1)(C)).
> **⚠️ Important:** § 1681s-2(a) duties are enforceable **only by government agencies (FTC, CFPB, state AGs)** — there is **no private right of action** for violations of subsection (a). Consumers cannot sue DRE for violating these duties directly. See *consumerprotection.net/furnisher-liability-fcra/*.
#### Duties under § 1681s-2(b) — Duty to Investigate Disputed Information (Upon Notice from CRA)
Once a CRA notifies DRE that a consumer has disputed information:
1. **Conduct an investigation** of the disputed information (§ 1681s-2(b)(1)(A))
2. **Review all relevant information** provided by the CRA (§ 1681s-2(b)(1)(B))
3. **Report results** to the CRA within 30 days (45 days for free annual report disputes) (§ 1681s-2(b)(1)(C))
4. **Report corrected information** to all nationwide CRAs if investigation finds inaccuracy (§ 1681s-2(b)(1)(D))
5. **Complete investigation promptly** and generally within 30 days (§ 1681s-2(b)(2))
> **⚠️ Important:** § 1681s-2(b) duties **DO create a private right of action**. Consumers can sue DRE for failing to properly investigate a dispute after being notified by a CRA. This is the primary FCRA liability risk for furnishers.
---
## 3. Can DRE Report B2B Debts to Consumer Credit Bureaus?
### 3.1 The Core Legal Question
When DRE collects a B2B debt — i.e., a debt owed by one business to another — can it report that debt to Equifax, Experian, or TransUnion consumer credit bureaus?
### 3.2 Corporate Debtors: No
- The FCRA defines **"consumer" as "an individual"** (15 U.S.C. § 1681a(c)).
- A corporation, LLC, partnership, or other business entity is **not a "consumer"** under the FCRA.
- The three nationwide consumer CRAs (Equifax, Experian, TransUnion) maintain credit files **only for individuals**.
- **Conclusion:** DRE **cannot** report a purely corporate B2B debt to consumer CRAs. These bureaus have no mechanism or authority to maintain a credit file for a business entity under FCRA.
### 3.3 Individual Sole Proprietors: Yes, with Caveats
- A sole proprietor who operates under a DBA (doing business as) is still an **individual** — a "consumer" under FCRA.
- DRE **can** report the debt against the individual's personal credit file.
- However, the reporting must:
- Identify the debt accurately as a business obligation (the CRA's Metro2 format supports business-purpose codes)
- Comply with all FCRA requirements (accuracy, dispute investigation, etc.)
- Comply with FDCPA requirements (validation notice, pre-reporting contact)
- Be accurately attributed to the individual (not the DBA entity)
### 3.4 Personal Guarantors: Yes
- If a business principal signed a **personal guarantee** on a B2B debt, that individual is personally liable.
- Under the FTC's Advisory Opinion to Tatelbaum (2001), a personal guarantor is a "consumer" for FCRA purposes, and the creditor/collector has permissible purpose to access and report on that individual's credit.
- DRE **can** report the debt against the personal guarantor's individual credit file.
- All FCRA and FDCPA obligations apply to the individual.
### 3.5 The FDCPA Distinction
The **Fair Debt Collection Practices Act (FDCPA)** — separate from FCRA — covers only debts "primarily for personal, family, or household purposes." **Commercial/B2B debts are explicitly excluded** from FDCPA coverage. See 15 U.S.C. § 1692a(5).
> **Practical impact:** When DRE collects a B2B debt from a corporation, the FDCPA's conduct restrictions (calling hours, harassment prohibitions, validation notice requirements) do not apply. However, if DRE reports a personal guarantor of a B2B debt to a consumer CRA, FCRA obligations still apply — and some courts have held that FDCPA can apply to guarantees where the guarantor is a consumer.
---
## 4. Legal Requirements Before Reporting a Consumer Debt
Under the **CFPB Debt Collection Rule** (12 CFR Part 1006, effective November 2021), before DRE can report a **consumer debt** to a credit bureau, it must:
### 4.1 Contact the Consumer First
DRE must take **at least one** of the following steps before reporting:
| Method | Detail |
|--------|--------|
| **In-person conversation** | Speak with the consumer in person |
| **Telephone conversation** | Speak with the consumer by phone |
| **Mail a letter** | Send a letter and wait a reasonable time (~14 days) without return as undeliverable |
| **Electronic communication** | Send an email/text and wait a reasonable time (~14 days) without bounce-back |
### 4.2 Provide a Validation Notice
If DRE sends a **validation notice** (required by FDCPA § 809 / 15 U.S.C. § 1692g) containing:
- The amount of the debt
- The name of the creditor
- A statement of the consumer's right to dispute the debt within 30 days
...this satisfies the "contact" requirement, and DRE may generally begin reporting to CRAs after sending it.
### 4.3 Do Not Report During the 30-Day Dispute Window
Under FDCPA § 809(b) (15 U.S.C. § 1692g(b)):
> If the consumer notifies the debt collector in writing within the thirty-day period described in subsection (a) that the debt, or any portion thereof, is disputed, the debt collector **shall cease collection of the debt, or any disputed portion thereof**, until the debt collector obtains verification of the debt.
The CFPB has interpreted "collection of the debt" to include **reporting to a credit bureau**. Therefore:
- If a consumer disputes the debt in writing within 30 days of receiving the validation notice
- DRE **must not report** the debt to a CRA until it obtains verification of the debt and mails it to the consumer
### 4.4 FCRA Does Not Have an Independent Pre-Reporting Notice Requirement
Unlike the FDCPA (which requires contact/validation before reporting), **FCRA itself does not require pre-reporting notice** to the consumer. The FCRA's notice requirements are triggered:
- **After** the consumer disputes (investigation duties under § 1681s-2(b))
- **When adverse action** is taken based on a CRA report (§ 1681m)
However, in practice, the FDCPA's pre-reporting rules usually govern DRE's timeline.
---
## 5. Dispute and Investigation Process
### 5.1 Direct Dispute (Consumer → Furnisher)
When a consumer directly disputes a debt to DRE:
1. DRE must note the dispute in its records
2. DRE **should not report** the debt as delinquent during the dispute investigation without noting it as disputed
3. DRE must conduct a reasonable investigation
4. DRE must correct any inaccuracies found
5. If DRE reports to CRAs, it must notify them of the dispute status
> **Enforcement:** No private right of action under § 1681s-2(a). Only CFPB/FTC/State AG can enforce.
### 5.2 CRA-Initiated Dispute (Consumer → CRA → Furnisher)
When a CRA notifies DRE that a consumer has disputed information:
1. **Trigger:** Notice of dispute from the CRA with all relevant information
2. **Investigation:** DRE must conduct a reasonable investigation within **30 days** (extendable to 45 days)
3. **Review:** DRE must review all relevant information provided by the CRA
4. **Report:** DRE must report results to the CRA
5. **Correction:** If inaccurate, DRE must correct with all nationwide CRAs
6. **Prevention:** DRE must not re-report inaccurate information
> **⚠️ Enforcement:** **Private right of action exists** under § 1681s-2(b). Consumers can sue DRE for failing to properly investigate.
### 5.3 Texas-Specific Dispute Process (Tex. Fin. Code § 392.202)
See [Section 6.3](#63-dispute-and-correction-procedure) below — Texas law adds a parallel dispute process with specific timelines that can be more restrictive than FCRA.
---
## 6. Texas-Specific Laws (Finance Code Chapter 392)
### 6.1 Scope: Consumer Debts Only
Texas Finance Code Chapter 392 defines:
- **"Consumer"** = an individual who has a consumer debt (§ 392.001(1))
- **"Consumer debt"** = an obligation primarily for **personal, family, or household purposes** (§ 392.001(2))
- **"Debt collection"** = actions in collecting **consumer debts** (§ 392.001(5))
> **Key finding:** Chapter 392 **does not apply to B2B debts**. It only covers debts owed by individuals for personal/family/household purposes. A corporate B2B debt falls entirely outside Chapter 392's scope.
However, if DRE is collecting a consumer debt (or a B2B debt with a personal guarantee from an individual), Chapter 392 governs DRE's conduct.
### 6.2 Surety Bond Requirement (§ 392.101)
DRE **must** have a **$10,000 surety bond** on file with the Texas Secretary of State **before engaging in debt collection** in Texas. This applies to all third-party debt collectors and credit bureaus.
- Bond must be issued by a surety company authorized to do business in Texas
- Copy of bond must be filed with the Secretary of State
- Bond is in favor of any person damaged by a Chapter 392 violation, plus the State of Texas
- Failure to file is a criminal offense (§ 392.402)
> **Verification:** Texas SOS Debt Collector Search — https://texas-sos.appianportalsgov.com/tpdc-public-search-portal
### 6.3 Dispute and Correction Procedure (§ 392.202)
Texas law provides a dispute process that **runs parallel** to the FCRA process. When an individual disputes the accuracy of an item in DRE's files:
#### Step 1: Notice of Inaccuracy (Written)
The individual notifies DRE in writing of the inaccuracy.
#### Step 2: DRE's Obligations
| Scenario | DRE's Obligation |
|----------|-----------------|
| **DRE does not report to CRAs** | Cease collection efforts until investigation determines accurate amount |
| **DRE does report to CRAs** | Investigate AND cease collection efforts until investigation determines accurate amount |
#### Step 3: Written Response (Within 30 Days)
DRE must send a written statement to the individual within **30 days**:
- (a) Denying the inaccuracy, OR
- (b) Admitting the inaccuracy, OR
- (c) Stating insufficient time to complete the investigation
#### Step 4: If DRE Admits Inaccuracy
- Correct the file within **5 business days**
- Immediately cease collection on the inaccurate portion
- Notify all prior recipients of the inaccurate report
#### Step 5: If More Time Needed
- Immediately change the file as the individual requested (provisional correction)
- Send corrected report to all prior recipients
- Cease collection efforts
- Upon completing investigation, inform individual of determination; if accurate, may resume reporting
> **⚠️ Texas Penalties:** Violation of Chapter 392 is a **misdemeanor** ($100–$500 fine per violation). Additionally, a violation is a **deceptive trade practice** under the Texas DTPA (Bus. & Comm. Code Ch. 17), which carries treble damages and attorney's fees. Individual consumers can sue for actual damages + $100 minimum per violation (§ 392.403(e)) + attorney's fees (§ 392.403(b)).
### 6.4 Texas FDCPA Comparison (Chapter 392 vs. Federal FDCPA)
| Aspect | Texas Finance Code § 392 | Federal FDCPA |
|--------|-------------------------|---------------|
| Scope | Consumer debts | Consumer debts (same definition) |
| Reporting to CRAs | Governs accuracy/disputes (§ 392.202) | Pre-reporting contact required (CFPB Rule) |
| Bond needed | $10,000 surety bond | None |
| Private right of action | Yes — actual damages + $100 minimum + attorney's fees | Yes — actual/statutory damages + attorney's fees |
| Criminal penalty | Yes — misdemeanor | No |
| DTPA treble damages | Yes (via § 392.404) | No |
---
## 7. Liability Risks for Incorrect Reporting
### 7.1 FCRA Liability
| Risk | Details |
|------|---------|
| **Furnisher investigation failure** | Private right of action under § 1681s-2(b) — failing to properly investigate a CRA-notified dispute. Actual damages, statutory damages ($100–$1,000), punitive damages, attorney's fees. |
| **Reporting inaccurate information** | Enforceable by CFPB/FTC/State AG. Potential civil penalties up to $3,500 per violation (CFPB) or $4,800 per violation (FTC) for unfair/deceptive acts. |
| **No permissible purpose** | If DRE obtains a consumer report without permissible purpose — $1,000 actual or statutory damages + punitive + attorney's fees. |
### 7.2 Texas Law Liability
| Risk | Details |
|------|---------|
| **Violation of Ch. 392** | Misdemeanor ($100–$500 fine), actual damages, **minimum $100 per violation** (§ 392.403(e)), attorney's fees |
| **DTPA claim** | Treble damages (up to three times economic damages) + mental anguish + attorney's fees |
| **Bond claim** | Consumer can claim against DRE's $10,000 surety bond |
### 7.3 FDCPA Liability (Consumer Debts Only)
For consumer debt reporting violations:
- Actual damages
- Statutory damages up to **$1,000**
- Class action damages up to **$500,000 or 1% of net worth**
- Attorney's fees and costs
- Debt validation violations: reporting before verification
### 7.4 State UDAP / Deceptive Trade Practices
Many states have their own unfair/deceptive trade practices laws. In Texas, a Chapter 392 violation is automatically a DTPA violation (§ 392.404). Other states may have similar "mirror" laws.
### 7.5 Reputational Risk
Even where legally permissible, reporting a corporate B2B debtor to a consumer credit bureau (via personal guarantee) can damage customer relationships. The individual's personal credit may be impacted for 7 years.
---
## 8. Alternative: Business Credit Bureaus
For purely B2B debts (corporate debtors without personal guarantees), DRE should use **business credit bureaus** instead of consumer CRAs.
### 8.1 Major Business Credit Bureaus
| Bureau | Key Product | What's Reported |
|--------|------------|----------------|
| **Dun & Bradstreet** | PAYDEX® Score (1–100), D&B Rating | Trade payment history, collection records, liens, judgments, UCC filings |
| **Experian Business** | Intelliscore Plus (1–100) | Business credit, trade payment data, public records |
| **Equifax Business** | Business Credit Risk Score | Payment history, trade lines, public records |
### 8.2 How to Report to Business Bureaus
**Dun & Bradstreet — Global Trade Exchange Program:**
- DRE can join the D&B Global Trade Exchange program as a data contributor
- Free to join; DRE reports trade payment experiences (positive and negative) on businesses
- Data is used to compute PAYDEX scores
- Requires establishing a Data Furnisher Agreement with D&B
- Reports are on the **business entity** (by D-U-N-S Number), not on individuals
- No FCRA restrictions apply because no "consumer" is involved
**Experian Business / Equifax Business:**
- DRE must establish a Data Furnisher Agreement with each bureau
- Each bureau has specific data submission requirements (typically Metro2 format or proprietary formats)
- Business credit data submission processes are generally less regulated than consumer reporting
### 8.3 Advantages of Business Credit Reporting
| Advantage | Detail |
|-----------|--------|
| **No FCRA restrictions** | Business credit reports are not "consumer reports" — FCRA does not apply |
| **No FDCPA restrictions** | Business debts are excluded from FDCPA |
| **No Texas Ch. 392 restrictions** | Chapter 392 only covers consumer debts |
| **Industry standard** | B2B creditors expect trade payment data on D&B |
| **Reporting positive payments** | Can report good payment history, not just delinquencies |
| **No 7-year limit** | Business credit data can be reported indefinitely |
### 8.4 Recommended Strategy for DRE
| Debt Type | Recommended Reporting Strategy |
|-----------|-------------------------------|
| Consumer debt (individual) | Report to Equifax, Experian, TransUnion under FCRA |
| B2B debt — corporate debtor | Report to D&B, Experian Business, Equifax Business |
| B2B debt — sole proprietor | Report to consumer CRAs (individual is the consumer) OR business bureaus |
| B2B debt — personal guarantor | Report to consumer CRAs (guarantor is a consumer under FCRA) |
| Mixed — corporate + personal guarantee | Report to business bureaus for the entity AND consumer CRAs for the guarantor |
---
## 9. Practical Recommendations for DRE
### 9.1 Before Reporting Any Debt
- [ ] Verify DRE has a **$10,000 surety bond** on file with the Texas Secretary of State (required by § 392.101)
- [ ] Verify DRE has written authorization and a **Data Furnisher Agreement** with each CRA it intends to report to
- [ ] Implement **written policies and procedures** for credit reporting accuracy (per FCRA § 1681s-2(a))
- [ ] Implement a **dispute handling process** that meets both FCRA § 1681s-2(b) and Texas Ch. 392 § 392.202 requirements
- [ ] Train staff on Metro2 reporting format compliance
### 9.2 When Debt Type Is Consumer (Individual, Personal/Family/Household)
1. Send **FDCPA validation notice** within 5 days of initial contact
2. Wait **30 days** or receive dispute resolution before reporting
3. Ensure debt is accurately documented (amount, original creditor, dates)
4. Report only accurate information
5. Follow FCRA furnisher rules for disputes
6. Follow Texas § 392.202 dispute process as applicable
### 9.3 When Debt Type Is B2B (Corporate)
1. **Do not report** to Equifax/Experian/TransUnion consumer CRAs
2. Consider reporting to **Dun & Bradstreet** via Global Trade Exchange
3. Consider reporting to **Experian Business** or **Equifax Business**
4. Note: No FCRA/FDCPA restrictions apply to business credit reporting
5. Consider including a **personal guarantee requirement** in future client contracts to enable consumer bureau reporting
### 9.4 When Debt Involves a Personal Guarantor
1. Confirm the guarantee is in writing and signed
2. Treat the guarantor as a consumer for FCRA purposes
3. Send validation notice to the guarantor
4. Follow all FCRA/Texas procedures before reporting
5. Report only the amount the guarantor is obligated for
### 9.5 Suggested Client Contract Language
Include in future DRE service agreements with creditor clients:
> *"Client represents that debts placed with DRE for collection may include consumer debts, commercial/B2B debts, and debts secured by personal guarantees. DRE may, at its discretion, report delinquent accounts to consumer credit reporting agencies (Equifax, Experian, TransUnion) and/or business credit reporting agencies (Dun & Bradstreet, Experian Business, Equifax Business), subject to all applicable legal requirements. Client agrees to provide DRE with all documentation necessary to verify any debt before reporting, including personal guarantee documents where applicable."*
---
## 10. Key Citations
### Federal Statutes
| Statute | Citation | Description |
|---------|----------|-------------|
| FCRA — Definitions | 15 U.S.C. § 1681a | Defines "consumer" as an individual, "consumer report," "consumer reporting agency" |
| FCRA — Permissible Purposes | 15 U.S.C. § 1681b | When consumer reports may be obtained |
| FCRA — Furnisher Duties | 15 U.S.C. § 1681s-2 | Accuracy, investigation, correction obligations |
| FCRA — Civil Liability | 15 U.S.C. § 1681n | Willful noncompliance: actual/statutory/punitive damages |
| FCRA — Civil Liability | 15 U.S.C. § 1681o | Negligent noncompliance: actual damages |
| FDCPA — Definitions | 15 U.S.C. § 1692a | Defines "debt" as personal/family/household only |
| FDCPA — Validation Notice | 15 U.S.C. § 1692g | Required notice contents; 30-day dispute period |
| CFPB Debt Collection Rule | 12 CFR Part 1006 | Pre-reporting contact requirements |
### Texas Statutes
| Statute | Citation | Description |
|---------|----------|-------------|
| Texas Fin. Code — Definitions | § 392.001 | "Consumer" = individual with consumer debt; "consumer debt" = personal/family/household |
| Texas Fin. Code — Bond | § 392.101 | $10,000 surety bond required |
| Texas Fin. Code — File Correction | § 392.202 | Dispute/verification/correction process |
| Texas Fin. Code — Civil Remedies | § 392.403 | Actual damages + $100 minimum per violation + attorney's fees |
| Texas Fin. Code — DTPA Remedy | § 392.404 | Violation = deceptive trade practice (treble damages available) |
| Texas Bus. & Comm. Code | Ch. 17 (DTPA) | Deceptive Trade Practices Act — treble damages |
### Advisory Opinions & Guidance
| Source | Document | Relevance |
|--------|----------|-----------|
| FTC Advisory Opinion | Tatelbaum (2001) | Personal guarantor on business loan = permissible purpose under FCRA |
| FTC Advisory Opinion | Gowen (1999) | Review of consumer report requires contractual authority to change terms |
| CFPB | Debt Collection Rule FAQs | Pre-reporting contact requirements |
| Texas SOS | FAQs for Third-Party Debt Collectors | Bond filing, registration, consumer complaints |
### Regulatory Bodies
| Entity | Role | Contact |
|--------|------|---------|
| **CFPB** | Enforces FCRA furnisher rules, FDCPA | consumerfinance.gov/complaint |
| **FTC** | Enforces FCRA, FDCPA | ftc.gov |
| **Texas SOS** | Bond filing, registry | (512) 475-0775 |
| **Texas OAG** | Consumer protection enforcement | texasattorneygeneral.gov |
| **Dun & Bradstreet** | Business credit data furnisher setup | dnb.com/global-trade-exchange |
---
*This document is for informational purposes and does not constitute legal advice. DRE should consult with a qualified attorney before implementing any credit reporting program.*
references/dre-fee-structure.md
+82
View File
@@ -0,0 +1,82 @@
# Debt Recovery Experts (DRE) — Fee Structure
## Proposed for review with Tony — July 2026
---
### Philosophy
Fee is based on effort required. Early resolution costs less, so clients pay less. Litigation requires attorney involvement, so the fee is higher. This incentivizes clients to provide good documentation upfront and rewards early recovery.
---
### Standard Fee Schedule
#### Tier 1 — Soft Touch (Automated email + payment link)
| Claim Amount | DRE Fee | Client Receives |
|---|---|---|
| $1,000 – $5,000 | 25% min $250 | 75% |
| $5,000 – $15,000 | 22% | 78% |
| $15,000+ | 20% | 80% |
*Includes: email demand, payment link (Stripe ACH), 14-day response window*
#### Tier 2 — Formal Demand (Certified mail)
| Claim Amount | DRE Fee | Client Receives |
|---|---|---|
| Any | 30% | 70% |
*Includes: Tier 1 + certified letter via LetterStream, 15-day demand notice, proof of delivery*
#### Tier 3 — Escalation (Final notice + intensive contact)
| Claim Amount | DRE Fee | Client Receives |
|---|---|---|
| Any | 33% | 67% |
*Includes: Tier 1-2 + final demand letter, additional communication attempts, enhanced skip tracing / debtor research*
#### Tier 4 — Legal Action (Referral to partner law firm)
| Component | DRE Fee | Law Firm Fee | Client Receives |
|---|---|---|---|
| Total fee | 10% referral | 25% litigation | 65% |
*DRE handles: case file prep, document transfer, client communication*
*Law firm handles: filing, court appearances, judgment enforcement*
---
### Example — $15,000 Claim
| Scenario | DRE Fee | Client Gets | Timeline |
|---|---|---|---|
| Paid at Tier 1 (email) | $3,300 (22%) | $11,700 | 2-14 days |
| Paid at Tier 2 (certified mail) | $4,500 (30%) | $10,500 | 15-30 days |
| Paid at Tier 3 (final notice) | $4,950 (33%) | $10,050 | 30-60 days |
| Resolved at Tier 4 (litigation) | $1,500 (10%) + $3,750 (25% law firm) | **$9,750** | 60-180 days |
---
### Out-of-Pocket Costs (passed to debtor or deducted at settlement)
| Cost | Amount | When |
|---|---|---|
| Online notarization (LPOA) | ~$25-35 | Per claim |
| Identity verification | ~$4 | Per signer |
| Certified mail (debtor) | ~$8.34 | Per letter |
| Court filing fees (if litigated) | ~$250-400 | Tier 4 only |
*Costs are either collected from the debtor as part of the recovery or deducted from the client's proceeds. DRE does not absorb these.*
---
### Recommendation
- **Start at 30% standard rate** for most claims (Tier 2)
- **Offer 20-25% for high-dollar claims** ($15K+) or repeat clients
- **Tier 4 referral model** (10% to DRE, 25% to firm) keeps it attractive for both DRE and the partner firm
- **Disclose all fees in LPOA** — Texas Finance Code requires clear disclosure
---
*Questions for discussion with Tony:*
- Flat 30% across all tiers vs tiered rates?
- Minimum fee floor ($250)?
- Volume discounts for repeat clients?
- Payment processing fees passed to debtor?
references/dre-letter-templates.md
+206
View File
@@ -0,0 +1,206 @@
# DRE Letter Templates — Reference
## AUTOMATED (Items 1-3)
### 1. Initial Onboarding Email
**From:** email@redacted
**To:** Client
**Subject:** We've Received Your Claim — DRE-2026-XXXX
> Thank you for submitting your claim with Debt Recovery Experts.
>
> Your claim has been received and logged. Here's what happens next:
>
> **Claim ID:** DRE-2026-XXXX
> **Debtor:** [Debtor Name]
> **Amount:** $X,XXX.00
>
> **Next Step:** Our AI system is reviewing your documentation. You'll hear from us within 1 business day with an assessment of your claim.
>
> In the meantime, you can upload additional documents at:
> portal.debtrecoveryexperts.com
>
> — Debt Recovery Experts
### 2. Claim Being Reviewed
**From:** email@redacted
**To:** Client
**Subject:** Claim DRE-2026-XXXX — Under Review
> Your claim is now being reviewed. Our team has begun analyzing the documentation you provided.
>
> **Claim ID:** DRE-2026-XXXX
> **Debtor:** [Debtor Name]
> **Status:** Under Review
>
> We're evaluating:
> • Documentation quality and completeness
> • Debtor profile and ability to pay
> • Best recovery strategy based on claim type and amount
>
> We'll follow up within 2-3 business days with our recommended next steps.
>
> — Debt Recovery Experts
### 3. Claim Accepted — LPOA Enclosed
**From:** email@redacted
**To:** Client
**Subject:** Claim DRE-2026-XXXX — Accepted + LPOA Attached
> Great news — your claim has been reviewed and accepted. We believe we can recover this debt.
>
> **Enclosed please find the Limited Power of Attorney (LPOA)** authorizing Debt Recovery Experts to act on your behalf in collecting this debt. Please:
>
> 1. **Review** the LPOA document
> 2. **Sign and notarize** (most banks offer free notary services)
> 3. **Return** via portal upload or reply to this email
>
> **What we need from you:**
> • Signed + notarized LPOA (attached)
> • Any additional documentation you'd like us to consider
>
> **What happens after LPOA is returned:**
> We send a soft-touch demand to the debtor. Most claims at this stage resolve without escalation.
>
> — Debt Recovery Experts
>
> **Attachments:** LPOA_DRE-2026-XXXX.pdf
---
## QUEUED — EDIT + APPROVAL REQUIRED (Items 4-8)
*These letters require: Anita drafts/edits → Tony & Germaine approve → Sent via email@redacted*
### 4. Soft Touch — 1st Email (Tier 1)
**From:** email@redacted
**To:** Debtor
**Subject:** Outstanding Balance — [Debtor Company]
> This is a friendly reminder regarding an outstanding balance with one of our clients.
>
> **Client:** [Client Name]
> **Amount Due:** $X,XXX.00
> **Invoice Reference:** [Invoice #]
>
> We understand things get overlooked. Please remit payment or contact us to discuss a resolution within 5 business days.
>
> Payment can be made at: pay.debtrecoveryexperts.com
>
> This communication is from a debt collector. Any information obtained will be used for that purpose.
>
> — Debt Recovery Experts
### 5. Formal Demand (Tier 2)
**From:** email@redacted
**To:** Debtor
**Subject:** FORMAL DEMAND FOR PAYMENT — [Debtor Company]
> This letter constitutes formal demand for full payment of the outstanding balance described below.
>
> **Client:** [Client Name]
> **Amount Due:** $X,XXX.00
> **Invoice Reference:** [Invoice #]
>
> Despite previous attempts to resolve this matter amicably, the amount remains unpaid.
>
> PLEASE TAKE NOTICE that if the full balance is not received within fourteen (14) calendar days, we will proceed with escalated collection measures, including but not limited to:
> • Referral to our legal department
> • Filing of a civil suit to obtain judgment
> • Placement of liens against real or personal property
>
> Contact our offices immediately to arrange payment or discuss a resolution.
>
> This communication is from a debt collector attempting to collect a debt. Any information obtained will be used for that purpose.
>
> — Debt Recovery Experts
> Collections Department
> email@redacted
### 6. Lien Threat (Tier 2.5 — Texas Construction)
**From:** email@redacted
**To:** Debtor
**Subject:** NOTICE OF INTENT TO FILE LIEN — [Property/Project]
> This letter serves as formal notice of our intent to file a lien against the property described below.
>
> **Client:** [Client Name]
> **Property/Project:** [Address or Description]
> **Unpaid Amount:** $X,XXX.00
>
> Texas Property Code allows for the filing of a mechanic's lien against real property where improvements were made and remain unpaid. We have been authorized to take the following actions:
>
> • Filing a Sworn Statement of Account with the county clerk
> • Recording a mechanic's lien against the property
> • Pursuing foreclosure on the lien if necessary
>
> A mechanic's lien will:
> • Attach to the property title
> • Affect your ability to sell or refinance
> • Appear on title searches
>
> To avoid lien filing, full payment must be received within ten (10) calendar days.
>
> This communication is from a debt collector attempting to collect a debt. Any information obtained will be used for that purpose.
>
> — Debt Recovery Experts
> Collections Department
### 7. Escalation — Final Notice (Tier 3)
**From:** email@redacted
**To:** Debtor
**Subject:** FINAL NOTICE — IMMEDIATE ACTION REQUIRED
> ⚠️ FINAL NOTICE — This is your last opportunity to resolve this matter before legal action.
>
> **Client:** [Client Name]
> **Amount Due:** $X,XXX.00
> **Original Invoice Date:** [Date]
>
> Multiple attempts have been made to collect this debt. Despite these efforts, the full balance remains unpaid.
>
> **UNLESS FULL PAYMENT IS RECEIVED WITHIN TEN (10) CALENDAR DAYS,** we will:
> • Refer this matter to our legal counsel
> • Initiate civil litigation to obtain a judgment
> • Pursue all available post-judgment remedies, including wage garnishment, bank account levy, and asset seizure
> • Report this debt to credit reporting agencies
>
> You may be held liable for court costs, attorney's fees, and additional interest.
>
> Contact our offices immediately. This is your final opportunity to resolve this without court intervention.
>
> This communication is from a debt collector attempting to collect a debt. Any information obtained will be used for that purpose.
>
> — Debt Recovery Experts
> Collections Department
### 8. Legal Action Notice (Tier 4)
**From:** email@redacted
**To:** Debtor
**Subject:** LEGAL ACTION — [Court/Claim Reference]
> This letter confirms that your account has been referred for legal action.
>
> **Client:** [Client Name]
> **Amount Due:** $X,XXX.00
> **Legal Reference:** DRE-2026-XXXX
>
> Effective immediately, this matter has been forwarded to our legal counsel for lawsuit preparation. A civil petition will be filed seeking:
>
> • Judgment for the full amount owed
> • Pre-judgment interest as allowed by law
> • Court costs and filing fees
> • Attorney's fees
>
> Upon obtaining a judgment, we will pursue collection through all available legal channels, including:
> • Wage garnishment (up to 25% of disposable earnings)
> • Bank account levy
> • Lien against real property
> • Post-judgment discovery of assets
>
> All further communication regarding this matter should be directed to our legal counsel.
>
> This communication is from a debt collector attempting to collect a debt. Any information obtained will be used for that purpose.
>
> — Debt Recovery Experts
> Legal Liaison Division
references/dre-roles-guide.md
+83
View File
@@ -0,0 +1,83 @@
# Debt Recovery Experts — Systems & Roles Guide
## Your Access Overview
### Anita — Read-Only
You can view claims, debtors, and case progress in the CRM. You cannot approve cases, make changes, or send emails from the system. This gives you full visibility into DRE operations without any responsibility for case decisions.
### Tony — Full Admin
You have the same level of access as Germaine. You can view, create, edit, and approve everything in the CRM. You can manage claims, debtors, payments, and case notes. You can approve cases and move them through recovery tiers. This is your business — you have full control.
---
## The Three Systems You'll Use
### 1. DRE CRM (TwentyCRM)
**https://crm.debtrecoveryexperts.com**
This is the central system where all claims, debtors, and case activity are tracked.
**What you'll find inside:**
- Claims — each recovery case with status, tier, amount, and debtor info
- Debtors — the companies and individuals we're collecting from
- Payments — money collected, fees deducted, net to client
- Case Notes — internal notes on case progress
Login: Click "Continue with Email" — a magic link is sent to your email. No password to remember.
---
### 2. DRE Portal
**https://portal.debtrecoveryexperts.com**
Public-facing pages for clients and debtors. You don't log into this — it's how the outside world interacts with DRE.
**Pages:**
- **Intake Form** — New clients submit claims here
- **Fee Calculator** — Clients can estimate fees
- **Payment Page** — Debtors pay here via ACH
---
### 3. Email — Three Mailboxes
| Address | Display Name | What It's For |
|---|---|---|
| **email@redacted** | Debt Recovery Experts | Debtor-facing. Demand letters, payment links, collection updates. |
| **email@redacted** | Debt Recovery Experts | Client-facing. Claim status updates, DocuSeal signing requests, fee receipts. |
| **email@redacted** | Debt Recovery Experts | General business inquiries. |
All three show "Debt Recovery Experts" as the sender name. The reply goes to whichever mailbox the client or debtor wrote to.
---
## How a Claim Flows
```
Client submits intake form
↓
Team reviews in CRM
↓
Germaine approves → moves to next tier
↓
Automated emails sent via collections@ or dre@
↓
Payment collected → recorded in CRM
↓
Case closed → binder generated
```
---
## Quick Reference
| Question | Answer |
|---|---|
| Where do I log in? | crm.debtrecoveryexperts.com |
| What's my login method? | Email magic link (no password) |
| Can I approve cases? | Tony: Yes — same as Germaine. Anita: No. |
| Can I add notes to a case? | Yes (if you can edit records) |
| Can clients see the CRM? | No — clients use the portal |
| Who do I contact for help? | Germaine |
references/dre-specs.md
+178
View File
@@ -0,0 +1,178 @@
# DRE Portal — Technical Specifications
## Internal working document — July 2026
### Claim Numbering
- Format: `DRE-YYYY-NNNN` (e.g. DRE-2026-0001)
- Auto-generated on claim submission
### Client ID
- Format: `CLT-YYYY-NNNN` (e.g. CLT-2026-0001)
- Generated on account creation
### Closed Case Binders
On case close, two PDF binders are auto-generated:
**Binder 1 — DRE Internal:**
DRE-YYYY-NNNN_CLOSED_DRE.pdf
Includes: intake, evidence, AI analysis, AI debtor research, AI weakness analysis,
internal notes, signed/notarized LPOA, certified mail receipts, communication log,
settlement, fee disclosure, payment confirmation
**Binder 2 — Client Package:**
DRE-YYYY-NNNN_CLOSED_Client.pdf
Includes: claim summary, signed/notarized LPOA, settlement/payment confirmation,
fee statement. EXCLUDES: AI analysis, debtor research, internal notes.
**Evidence Archive:**
DRE-YYYY-NNNN_evidence.zip — original uploaded documents
### Full Workflow
#### Phase 1 — Intake
- Client submits claim at debtrecoveryexperts.com
- Intake form: debtor info, amount, description
- Document upload (contracts, invoices, correspondence)
- Terms of Service signed via DocuSeal
- Claim enters DRE queue
#### Phase 2 — AI Review
- **AI analysis is per claim, not per client.** Each claim gets its own independent:
- Claim analysis score (0-100)
- Debtor research / skip tracing
- Weakness assessment
- If a repeat client submits a new claim against a different debtor, it gets fresh analysis independent of previous claims
- AI Re-analysis on new document upload is scoped to that specific claim only
- Team reviews → Approve / Request More Docs / Reject
#### Phase 3 — Legal Setup
- LPOA sent to client via DocuSeal
- Client signs → notarized via Proof (online RON)
- DRE now authorized to collect
#### Phase 4 — Tiered Recovery
- Tier 1: Soft Touch — email + ACH link (Day 1-5)
- Tier 2: Formal Demand — certified mail via LetterStream (Day 7-14)
- Tier 2.5: Lien Threat — pre-lien notice for construction claims (Day 15-21)
- Tier 3: Escalation — final notice (Day 21-30)
- Tier 4: Legal Action — referral to partner law firm (Day 30+)
#### Phase 5 — Settlement
- Payment collected via Stripe ACH
- DRE fee deducted per tier schedule
- Costs deducted (notary, certified mail, filing fees)
- Balance disbursed to client
- Case closed → two binders generated
### Fee Structure (proposed)
- Tier 1: 20-25%
- Tier 2: 30%
- Tier 2.5: 30% (+ attorney if lien filed)
- Tier 3: 33%
- Tier 4: 10% DRE referral + 25% law firm
### Search & Document Retrieval
Cases are searchable in the portal via database fields:
- Claim number, client name, debtor name, amount, status, date range
- Tags (e.g. "construction", "lien filed", "litigated", "debtor paid Tier 2")
- Key flags from AI analysis
On case close, PDF text is extracted and stored in the database for full-text search.
Future: Paperless-ngx if volume exceeds 500+ cases.
### Approval Workflow
- **Approvers:** Germaine + Tony — both must sign off before case moves forward
- **Anita:** Read-only access — can view pending approvals but cannot approve
- **Reminders:** Every 60 minutes, pending approval notification sent to both approvers
- Approval status visible in claim header: "Pending Germaine · Pending Tony · Acknowledged ✓"
Any edit to a claim submission is tracked with a full history:
- What field was changed (old value → new value)
- Who made the change (user name or "system" for automated updates)
- When the change was made (timestamp)
- Why the change was made (optional note field)
The audit log is visible in the DRE internal dashboard under each claim:
```
DRE-2026-0042 — Change History
Jul 8, 2026 10:15 AM | John (DRE) | Amount: $12,000 → $12,450 | "Corrected from invoice"
Jul 7, 2026 02:30 PM | System | Status: Submitted → AI Review | Auto
Jul 7, 2026 02:28 PM | Robert Smith | Initial submission |
```
Edits are allowed on open claims. Closed claims are read-only.
The audit trail is included in the DRE Internal Binder on case close.
### Repeat Clients
Clients create an account on first claim. On return:
- ToS already on file (signed once)
- Client info pre-filled on new claims
- Dashboard shows: active cases, past cases with outcomes, payment history
- Can download their own case binders
- Optional loyalty pricing after 3+ claims (e.g. 25% vs 30%)
### Client Portal View (after login)
- Active Cases: claim number, debtor, amount, current tier
- Past Cases: claim number, debtor, amount, outcome (Recovered / Uncollectible)
- [Submit New Claim] button
- [Download Case Binders] for closed cases
- [Payment History] — all disbursements received
### Required Services
- DocuSeal: ✅ deployed at sign.itpropartner.com
- Proof.com: ❌ need account (online notarization)
- LetterStream: ❌ need account (certified mail)
- Stripe Connect: ❌ need keys (ACH payments)
- Law firm partner: ❌ need referral agreement (litigation)
### Email Configuration
- **email@redacted** — Client-facing. Display name: "Debt Recovery Experts". Used for DocuSeal signing requests, claim status updates, fee receipts, portal login links.
- **email@redacted** — Debtor-facing. Display name: "Debt Recovery Experts". Used for demand letters, ACH payment links, collection correspondence.
- **email@redacted** — General business inquiries.
- All mailboxes hosted on MXroute (heracles.mxrouting.net). Outgoing email relayed through mail.germainebrown.com:2525 (MXroute SMTP blocks our IP).
### AI Chatbot — Knowledge Base
The DRE chatbot runs on Llama 3.2 3B (local, Ollama on Core) with a structured FAQ knowledge base built once by a smart model (Claude Opus 4.7).
**Chatbot Personality:**
- **Empathetic** — Acknowledge the frustration of an unpaid debt. "I understand how stressful unpaid invoices can be."
- **Kind but firm** — Supportive without being a pushover. Never apologetic for DRE's process or fees.
- **Engaging** — Ask leading questions to qualify and move the conversation forward. "How long has this been outstanding?"
- **No promises** — Never guarantee a specific outcome, timeline, or recovery amount. Use "typically," "most claims," "can be."
- **Escalate when stuck** — If the visitor asks something outside the bot's scope, invite them to submit a claim or contact the team.
**Tone examples:**
| Do | Don't |
|---|---|
| "I understand — unpaid invoices create real pressure." | "Sorry about that." |
| "Most claims at your level resolve within 30 days." | "We'll get your money back in 30 days." |
| "Our process typically recovers 70%+ of claims." | "You'll definitely get paid." |
| "Ready to get started? We can review your case." | "You should definitely use us." |
**PUBLIC knowledge (chatbot can answer):**
- Fee structure & tier percentages
- Required documents for a claim
- Recovery timelines per tier
- Texas construction lien overview (general)
- LPOA process (what it is, how to sign)
- Payment/ACH processing info
- How to submit a claim
- Contact/support info
- Link to Fee Calculator
- Link to Terms of Service & Privacy Policy
**PRIVATE knowledge (NEVER in chatbot):**
- AI claim scoring methodology
- Debtor research / skip tracing techniques
- Internal weakness analysis criteria
- Approval workflow and review criteria
- Specific claim details or case data
- Partner law firm names or referral terms
- Internal fee negotiation limits
- FDCPA/TDCA compliance strategy
- Any internal team notes or communications
### Future Automation (N8n on Monitor Server)
**AI Web Scraper Agent** — https://n8n.io/workflows/2006-ai-agent-that-can-scrape-webpages/
- 4.6/5 rating, free workflow template
- ReAct AI agent that scrapes web pages for skip tracing
- **Use cases:** Texas SOS lookups, county court records, debtor asset searches
- **When:** Remind Germaine about this when building DRE portal — can offload debtor research to n8n workflows
references/dre-temp-passwords.txt
+3
View File
@@ -0,0 +1,3 @@
Germaine: rRaWz9-N5VC7I-2ZOUQR
Tony: s9o9ap-pEHDvV-3GwF2y
Anita: Ekuchg-Nhc9Ju-6wPDuZ
references/hetzner-netcup-migration-plan.md
+170
View File
@@ -0,0 +1,170 @@
# Hetzner to netcup Migration Plan
Author: Sho'Nuff
Last updated: 2026-07-09
Status: PLAN (not yet executed)
Sources: app-inventory.csv (Germaine's Migrate/New Host columns), hetzner-server-audit.md,
server-provisioning-standard, ops-portal-and-collector skill.
---
## Objective
Consolidate the ITPP server fleet by moving apps/services off Hetzner (whose new-server
pricing is now premium) onto netcup root (dedicated) servers. Goal: fewer servers, lower
recurring cost, and NOT an all-in-one design -- workloads are spread so a single outage
does not take down everything and everyone.
Guiding constraints from Germaine (Jul 9, 2026):
- Cost savings is good but NOT the priority; plenty of overhead for growth on each box.
- Do not put everything on one server (blast-radius isolation).
- Keep the MikroTik router API functionality.
- Keep Ollama on core as a local fallback if admin-ai connectivity is lost.
- admin-ai (LiteLLM) must stay online at all times -- it is where Hermes gets its model.
- app1-bu stays as the warm standby at Hetzner (do NOT migrate).
---
## Target Architecture
Base hardware standard for all NEW netcup servers: RS 4000 G12
- 12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe
- ~$44/mo (monthly billing, Manassas VA / Virginia hosting)
- Monthly billing chosen (NOT annual).
Server roles after migration:
Core (existing, netcup RS 2000 G12, 152.53.192.33) -- KEEP AS-IS
- Role: Ops hub. Hermes, Caddy reverse proxy, ops portal, Docker services
(Twenty CRM, Vaultwarden, DocuSeal, SearXNG), Shark Game, Prometheus, Ollama (fallback), VPN.
- Also receives: Uptime-Kuma (migrated from Hetzner 'docker' box).
app1 (NEW, netcup RS 4000 G12) -- AI / ML + automation
- LiteLLM (admin-ai), Open WebUI, Ollama (primary), Qdrant, PostgreSQL (LiteLLM backend), Caddy
- n8n (workflow automation) + its PostgreSQL -- placed here so it can use local Ollama
- Follows the /docker/<service> install standard (current Hetzner install does NOT).
app2 (NEW, netcup RS 4000 G12) -- Infrastructure / WISP / network
- UNMS/UISP (10-container stack), UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd
- UniFi Controller + MongoDB + Postfix
- Hudu docs + SWAG + PostgreSQL + Redis
- Traccar (fleet GPS) + Apache frontend
app3 (NEW, netcup RS 4000 G12) -- Web hosting
- All WordPress sites currently on wphost02 (Apex Track Experience + others)
- MariaDB, Postfix, PHP-FPM
- New-site provisioning workflow (replacing RunCloud stand-up) lives here.
app1-bu (existing, Hetzner CPX11, 5.161.114.8) -- KEEP AS WARM STANDBY
- Stays at Hetzner. Warm standby clone of Hermes, always on, gateway dormant.
- Failover: checks Core every 5 min; if Core down, confirms over 2 min then activates gateway.
---
## Per-Service Migration Map
Source -> Destination (from app-inventory.csv "Migrate / New Host" columns):
From ai.itpropartner.com (Hetzner CPX41, 178.156.167.181) -> app1
- LiteLLM -> app1 (reinstall under /docker standard)
- Open WebUI -> app1 (conversations are in Postgres/DB -- migrate DB, not just app)
- Ollama (remote) -> app1
- Qdrant -> app1 (confirm whether actually wired to OWUI/LiteLLM before migrating)
- Caddy (ai) -> app1
- PostgreSQL (ai) -> app1 (LiteLLM uses DB method, not config file -- migrate the DB)
NOTE: This box is at 92% disk -- migrate before it fills. admin-ai must not have downtime;
stand up new LiteLLM on app1, cut DNS over, then decommission old.
From hudu.itpropartner.com (Hetzner CPX21, 178.156.130.130) -> app2
- Hudu Docs, SWAG, PostgreSQL, Redis, MariaDB, Postfix
From unms.forefrontwireless.com (Hetzner CPX21, 5.161.225.131) -> app2
- UNMS/UISP, UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd, MariaDB
- Full WISP management stack -- migrate existing app/service/DB to new host.
From unifi (Hetzner CPX21, 178.156.131.57) -> app2
- UniFi Controller + MongoDB + Postfix
- MariaDB (UniFi): NOT migrated (N/A).
From wphost02 (Hetzner CPX21, 5.161.62.38) -> app3
- WordPress (Apex) + all other WordPress sites hosted here, MariaDB, Postfix, PHP-FPM
- All WordPress sites must be transferred. SiteGround SFTP backups already captured (21 sites).
From app1.itpropartner.com (Hetzner CPX11, 87.99.144.163) -> app1
- n8n + PostgreSQL. Migrate automations AND saved credentials.
- Nginx reverse proxy for n8n. MariaDB (n8n): NOT migrated (N/A).
From docker (Hetzner CPX11, 178.156.168.35)
- Uptime-Kuma -> core (migrate app + data)
- Nginx Proxy Manager -> NOT migrated (N/A)
- RustDesk -> NOT migrated (N/A)
- StrongSwan (Docker) -> NOT migrated (N/A)
- RunCloud PHP -> NOT migrated (N/A)
NOTE: 73% disk. Once Uptime-Kuma is moved, this box can be freed.
From fleettracker360 (Hetzner CPX11, 178.156.149.32) -> app2
- Traccar (native Java) + Apache frontend. MariaDB: NOT migrated (N/A).
Stays in place (no migration):
- app1-bu (warm standby) -- remains at Hetzner.
- Home Router Keepalive / IPsec on app1-bu -- functionality already exists on core.
---
## Cost Summary (USD, monthly)
Current relevant Hetzner spend is being replaced by 3 netcup RS 4000 boxes.
- app1 (RS 4000): ~$44/mo
- app2 (RS 4000): ~$44/mo
- app3 (RS 4000): ~$44/mo
- Core (RS 2000): ~$24/mo (unchanged)
- app1-bu (CPX11): ~$8/mo (unchanged, stays at Hetzner)
Servers freed at Hetzner after migration: ai, hudu, unms, unifi, wphost02, app1(n8n),
docker, fleettracker360 (8 boxes) -- decommission after verification, avoiding Hetzner's
new premium pricing on any future provisioning.
---
## Migration Sequence (recommended, safety-first)
Precondition: full audit GREEN and all backups verified before starting (Germaine's rule).
Also fix DR-006 (schedule the full Hermes backup) BEFORE decommissioning anything.
Phase 0 -- Provision
1. Order app1, app2, app3 (RS 4000 G12, monthly, Virginia) via netcup SCP.
2. Base-harden each per server-provisioning-standard (SSH keys, firewall, Docker under /docker).
Phase 1 -- app1 (AI/ML) FIRST, because admin-ai is critical
3. Stand up LiteLLM + Postgres on app1 under /docker standard; import the LiteLLM DB.
4. Verify Hermes can reach the new admin-ai endpoint (test deepseek-chat) BEFORE cutover.
5. Move Open WebUI (migrate DB so conversations transfer), Ollama, Qdrant.
6. Move n8n + Postgres + credentials.
7. Cut DNS (admin-ai.itpropartner.com, ai.itpropartner.com, app1.itpropartner.com) to app1.
8. Verify, then decommission old ai + app1(n8n) Hetzner boxes.
Phase 2 -- app2 (Infrastructure/WISP)
9. Migrate UNMS/UISP + UCRM stack, UniFi + MongoDB, Hudu, Traccar (apps + services + DBs).
10. Cut DNS, verify each, decommission unms/unifi/hudu/fleettracker360.
Phase 3 -- app3 (Web)
11. Replicate WordPress hosting structure; transfer all sites + MariaDB from wphost02.
12. Build new-site provisioning workflow (RunCloud replacement) on app3.
13. Cut DNS, verify all sites, decommission wphost02.
Phase 4 -- Core cleanup
14. Migrate Uptime-Kuma to core; free the Hetzner 'docker' box.
Phase 5 -- Post-migration audit
15. Run a full DR audit again; confirm all backups, services, DNS, and failover healthy.
---
## Open Items / Risks
- IAM key lacks s3:CreateBucket -- new S3 buckets (system-configs, docker-volumes) must be
created in the Wasabi Console before their sync scripts can write (see dr-issue-log DR-011).
- Qdrant may not currently be wired to Open WebUI / LiteLLM -- confirm before migrating.
- admin-ai has zero-downtime requirement -- use stand-up-then-cutover, never in-place move.
- Docker installs on the Hetzner AI box do NOT follow the /docker/<service> standard;
the new app1 install must follow it.
references/hetzner-server-audit.md
+839
View File
@@ -0,0 +1,839 @@
# Hetzner Server Inventory Audit
> **Live document.** Comprehensive SSH-based audit of every running server.
> Last updated: 2026-07-09
> All servers SSH'd via `root@<ip>` with `itpp-infra` key.
---
## Quick Summary (10 Servers)
| # | Server | Hostname | IP | Specs | OS | Docker? | Key Apps |
|---|--------|----------|----|-------|----|---------|----------|
| 1 | **app1-bu** | app1-bu.itpropartner.com | 5.161.114.8 | 2C/2G/40G | Ubuntu 26.04 | ❌ | Standby Hermes, StrongSwan VPN, L2TP |
| 2 | **ai.itpropartner.com** | ai | 178.156.167.181 | 8C/16G/240G | Ubuntu 24.04 | ✅ 6 containers | LiteLLM, Open WebUI, Ollama, Qdrant, Caddy |
| 3 | **hudu.itpropartner.com** | hudu | 178.156.130.130 | 3C/4G/80G | Ubuntu 24.04 | ✅ 5 containers | Hudu Docs, SWAG (Let's Encrypt), MariaDB, Postfix |
| 4 | **unifi** | unifi | 178.156.131.57 | 3C/4G/80G | Ubuntu 24.04 | ❌ | UniFi Controller, MongoDB, MariaDB |
| 5 | **unms.forefrontwireless.com** | unms | 5.161.225.131 | 3C/4G/80G | Ubuntu 24.04 | ✅ 9 containers | UNMS/UISP, UCRM, Nginx, RabbitMQ, PostgreSQL |
| 6 | **wphost02** | wphost02 | 5.161.62.38 | 3C/4G/80G | Ubuntu 24.04 | ❌ | RunCloud (Nginx, Apache, MariaDB, WordPress) |
| 7 | **app1.itpropartner.com (n8n)** | n8n | 87.99.144.163 | 2C/2G/40G | Ubuntu 24.04 | ✅ 2 containers | n8n, PostgreSQL, Nginx, MariaDB |
| 8 | **docker** | docker | 178.156.168.35 | 2C/2G/40G | Ubuntu 24.04 | ✅ 7 containers | NPM, RustDesk, Uptime-Kuma, StrongSwan, RunCloud |
| 9 | **fleettracker360** | fleettrack360 | 178.156.149.32 | 2C/2G/40G | Ubuntu 24.04 | ❌ | Traccar, RunCloud (Apache/Nginx), MariaDB |
| 10 | **app1 (netcup - live Hermes)** | core | 152.53.192.33 | 8C/15G/512G | Debian 13 | ✅ 6 containers | Hermes, Twenty CRM, Vaultwarden, Docuseal, SearXNG, Caddy |
---
## 1. 🔵 app1-bu — Hermes Standby
| Field | Value |
|-------|-------|
| **IP** | 5.161.114.8 |
| **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 |
| **Specs** | 2 vCPU AMD EPYC-Rome · 1.9 GB RAM · 38 GB disk |
| **OS** | Ubuntu 26.04 LTS (Resolute Raccoon) |
| **Kernel** | 7.0.0-27-generic |
| **Uptime** | 3 days 13:46 |
| **Load** | 0.01 / 0.03 / 0.01 |
| **Disk** | 54% used (20G of 38G) |
### Docker
❌ **No Docker installed**
### Running Systemd Services (custom)
- `hermes-gateway.service` — Hermes Gateway (disabled/dormant)
- `hermes-standby.service` — Standby restore service
- `home-router-keepalive.service` — Router keepalive
- `home-router-vpn.service` — Router VPN tunnel
- `strongswan-starter.service` — IPsec IKEv1/IKEv2
- `xl2tpd.service` — L2TP tunnel daemon
### Listening Ports
| Port | Service |
|------|---------|
| 22 | SSH |
| 53 | systemd-resolved (DNS) |
| 500 | charon (IPsec IKE) |
| 4500 | charon (IPsec NAT-T) |
| 1701 | xl2tpd (L2TP) |
| 323 | chronyd (NTP) |
### Web Servers
❌ None
### Databases
❌ None
### Users
- `root` (only user)
### Cron (root)
```cron
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh
```
### Custom Scripts
- `/root/triage_continue.sh`
- `/root/triage_run.sh`
- `/root/triage_verify_summary.py`
---
## 2. 🔵 ai.itpropartner.com — AI Proxy
| Field | Value |
|-------|-------|
| **IP** | 178.156.167.181 |
| **Hostname** | ai |
| **Plan** | CPX41 |
| **Specs** | 8 vCPU AMD EPYC-Rome · 15 GB RAM · 226 GB disk |
| **OS** | Ubuntu 24.04.4 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 19:59 |
| **Load** | 0.02 / 0.06 / 0.08 |
| **Disk** | ⚠️ **92% used** (199G of 226G) |
### Docker Containers (6 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `litellm` | ghcr.io/berriai/litellm:v1.84.0 | Up 3 days (unhealthy) | 4000 |
| `litellm_postgres` | postgres:16 | Up 3 days (healthy) | 5432 |
| `openwebui` | ghcr.io/open-webui/open-webui:0.9.6 | Up 3 days (healthy) | 8080 |
| `ollama` | ollama/ollama:latest | Up 3 days | 11434 |
| `qdrant` | qdrant/qdrant:latest | Up 3 days | 6333-6334 |
| `caddy` | caddy:latest | Up 3 days | 80, 443 |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| ollama/ollama | latest | 10.1 GB |
| ghcr.io/open-webui/open-webui | 0.9.6 | 6.73 GB |
| ghcr.io/open-webui/open-webui | main | 6.7 GB |
| ghcr.io/berriai/litellm | v1.84.0 | 2.58 GB |
| postgres | 16 | 641 MB |
| qdrant/qdrant | latest | 285 MB |
| caddy | latest | 87.9 MB |
| curlimages/curl | latest | 36.1 MB |
### Docker Compose
- `/opt/ai/docker-compose.yml`
- `/opt/ai/docker-compose-last.yml`
### Running Systemd Services (non-standard)
- `containerd.service`, `docker.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 80 | HTTP → Caddy (Docker) |
| 443 | HTTPS → Caddy (Docker) |
| 4000 | LiteLLM |
| 6333-6334 | Qdrant |
| 11434 | Ollama |
### Web Servers
❌ Native — Caddy runs in Docker
### Databases
❌ None native (PostgreSQL runs in Docker)
### Users
- `root`, `ippadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 3. 🔵 hudu.itpropartner.com — Documentation
| Field | Value |
|-------|-------|
| **IP** | 178.156.130.130 |
| **Hostname** | hudu |
| **Plan** | CPX21 |
| **Specs** | 3 vCPU AMD EPYC-Rome · 3.7 GB RAM · 75 GB disk |
| **OS** | Ubuntu 24.04.4 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:44 |
| **Load** | 0.00 / 0.02 / 0.00 |
| **Disk** | 23% used (16G of 75G) |
### Docker Containers (5 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `hudu2-app-1` | hududocker/hudu:latest | Up 3 days | 3000 |
| `hudu2-worker-1` | hududocker/hudu:latest | Up 3 days | 3000 |
| `hudu2-db-1` | postgres:16.2 | Up 3 days | 5432 |
| `hudu2-redis-1` | redis:latest | Up 3 days | 6379 |
| `letsencrypt` | lscr.io/linuxserver/swag | Up 3 days | 80, 443 |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| hududocker/hudu | latest | 3.64 GB |
| lscr.io/linuxserver/swag | latest | 602 MB |
| postgres | 16.2 | 611 MB |
| redis | latest | 202 MB |
| alpine | latest | 12.8 MB |
| hello-world | latest | 20.3 kB |
### Running Systemd Services (non-standard)
- `containerd.service`, `docker.service`
- `fail2ban.service`, `firewalld.service`
- `mariadb.service` — **MariaDB 11.4.12** (native, not Docker)
- `email@redacted` — Postfix mail server
- `runcloud-agent.service` — RunCloud agent
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service`, `nginx-rc.service`
- `php81rc-fpm.service` through `php85rc-fpm.service`
- `runcloud-admin.service`, `runcloud-agent.service`, `runcloud-upgrade.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80 | HTTP → SWAG (Docker) |
| 443 | HTTPS → SWAG (Docker) |
| 3306 | MariaDB |
| 34210 | RunCloud |
### Databases
- **MariaDB** 11.4.12 (native, active)
- **MySQL** CLI (mysql/mysqld)
- **Redis** (native, active)
### Web Servers
❌ Native — SWAG (Nginx) runs in Docker for Let's Encrypt
- RunCloud custom Apache/Nginx installed but not running
### Users
- `root`, `runcloud`, `itpropartner`, `ippadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 4. 🔵 unifi — Network Controller
| Field | Value |
|-------|-------|
| **IP** | 178.156.131.57 |
| **Hostname** | unifi |
| **Plan** | CPX21 |
| **Specs** | 3 vCPU AMD EPYC-Rome · 3.7 GB RAM · 75 GB disk |
| **OS** | Ubuntu 24.04.3 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:44 |
| **Load** | 0.06 / 0.02 / 0.00 |
| **Disk** | 14% used (9.6G of 75G) |
### Docker
❌ **No Docker installed**
### Running Systemd Services (non-standard)
- `unifi.service` — **UniFi Network Controller** (Java)
- `snapd.service` — Snap daemon (certbot)
- `fail2ban.service`, `firewalld.service`
- `mariadb.service` — MariaDB 11.4.12
- `email@redacted` — Postfix mail server
- `runcloud-agent.service`
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service`, `nginx-rc.service`
- `php81rc-fpm.service` through `php85rc-fpm.service`
- `runcloud-admin.service`, `runcloud-agent.service`, `runcloud-upgrade.service`
- `snap.certbot.renew.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 3306 | MariaDB |
| 8080 | UniFi Controller (HTTP) |
| 8443 | UniFi Controller (HTTPS) |
| 8880 | UniFi (HTTP portal redirect) |
| 8843 | UniFi (HTTPS portal redirect) |
| 6789 | UniFi (Speed Test / throughput) |
| 27117 | MongoDB (local) |
| 34210 | RunCloud |
| 3478 | STUN (UDP) |
| 5514 | Syslog (UDP) |
| 10001 | UBNT discovery (UDP) |
### Databases
- **MariaDB** 11.4.12 (native, active)
- **MongoDB** (native, running on 127.0.0.1:27117)
- **Redis** (installed)
- MySQL CLI
### Web Servers
❌ Native — UniFi has built-in web server
- RunCloud custom Apache/Nginx installed but not running
### Users
- `root`, `runcloud`, `itpropartner`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 5. 🔵 unms.forefrontwireless.com — WISP Management
| Field | Value |
|-------|-------|
| **IP** | 5.161.225.131 |
| **Hostname** | unms |
| **Plan** | CPX21 |
| **Specs** | 3 vCPU AMD EPYC-Rome · 3.7 GB RAM · 75 GB disk |
| **OS** | Ubuntu 24.04.4 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:44 |
| **Load** | 0.61 / 0.55 / 0.38 |
| **Disk** | 67% used (48G of 75G) |
### Docker Containers (9 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `unms-nginx` | ubnt/unms-nginx:3.0.140 | Up 3 days | 80, 81, 443, 8089 |
| `unms-api` | ubnt/unms:3.0.140 | Up 3 days (healthy) | — |
| `unms-device-ws-1` | ubnt/unms:3.0.140 | Up 3 days (healthy) | — |
| `unms-device-ws-2` | ubnt/unms:3.0.140 | Up 3 days (healthy) | — |
| `unms-postgres` | ubnt/unms-postgres:3.0.140 | Up 3 days | 5432 |
| `unms-siridb` | ubnt/unms-siridb:3.0.140 | Up 3 days (healthy) | 9000, 9010 |
| `unms-rabbitmq` | rabbitmq:3.7.28-alpine | Up 3 days | 4369, 5671-5672, 25672 |
| `unms-fluentd` | ubnt/unms-fluentd:3.0.140 | Up 3 days | 5140, 24224 |
| `unms-netflow` | ubnt/unms-netflow:3.0.140 | Up 3 days | 2055/udp |
| `ucrm` | ubnt/unms-crm:4.5.31 | Up 3 days | 80-81, 443, 9000, 2055/udp |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| ubnt/unms | 3.0.140 | 1.07 GB |
| ubnt/unms-crm | 4.5.31 | 918 MB |
| ubnt/unms-postgres | 3.0.140 | 833 MB |
| ubnt/unms-nginx | 3.0.140 | 227 MB |
| ubnt/unms-netflow | 3.0.140 | 213 MB |
| ubnt/unms-siridb | 3.0.140 | 75.4 MB |
| ubnt/unms-fluentd | 3.0.140 | 39.5 MB |
| ubnt/ucrm-conntrack | latest | 6.29 MB |
| rabbitmq | 3.7.28-alpine | 102 MB |
| hello-world | latest | 10.1 kB |
### Running Systemd Services (non-standard)
- `containerd.service`, `docker.service`
- `fail2ban.service`
- `mariadb.service` — MariaDB 11.4.12 (native)
- `email@redacted`
- `runcloud-agent.service`
- `snapd.service`
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service`, `nginx-rc.service`
- `php81rc-fpm.service` through `php85rc-fpm.service`
- `runcloud-admin.service`, `runcloud-agent.service`, `runcloud-upgrade.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80, 81 | HTTP → UNMS Nginx (Docker) |
| 443 | HTTPS → UNMS Nginx (Docker) |
| 3306 | MariaDB (native) |
| 8089 | UNMS Nginx (alt) |
| 2055/udp | UNMS Netflow |
| 24224 | Fluentd (local) |
| 34210 | RunCloud |
### Databases
- **MariaDB** 11.4.12 (native, active)
- **Redis** (native, active)
- MySQL CLI
### Web Servers
❌ Native — Nginx/Apache via RunCloud installed but not primary
- UNMS Nginx runs in Docker as main frontend
### Users
- `root`, `runcloud`, `unms`, `fwadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 6. 🔵 wphost02 — WordPress Host
| Field | Value |
|-------|-------|
| **IP** | 5.161.62.38 |
| **Hostname** | wphost02 |
| **Plan** | CPX21 |
| **Specs** | 3 vCPU AMD EPYC-Rome · 3.7 GB RAM · 75 GB disk |
| **OS** | Ubuntu 24.04 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:45 |
| **Load** | 0.12 / 0.10 / 0.07 |
| **Disk** | 19% used (14G of 75G) |
### Docker
❌ **No Docker installed**
### Running Systemd Services (non-standard)
- `apache2-rc.service` — **Apache HTTP Server** (RunCloud)
- `nginx-rc.service` — **Nginx reverse proxy** (RunCloud)
- `php81rc-fpm.service` — PHP-FPM 8.1
- `mariadb.service` — MariaDB 11.1.6
- `email@redacted`
- `fail2ban.service`, `firewalld.service`
- `runcloud-agent.service`
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service`, `nginx-rc.service`
- `php81rc-fpm.service` through `php85rc-fpm.service`
- `runcloud-admin.service`, `runcloud-agent.service`, `runcloud-upgrade.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80 | HTTP → Nginx (RunCloud) |
| 443 | HTTPS → Nginx (RunCloud) |
| 81 | Apache (local proxy target) |
| 3306 | MariaDB |
| 34210 | RunCloud |
### Web Servers
- **nginx-rc** (Nginx w/ RunCloud — active frontend)
- **apache2-rc** (Apache w/ RunCloud — backend on port 81)
### Databases
- **MariaDB** 11.1.6 (native, active)
- **Redis** (native, active)
- MySQL CLI
### Users
- `root`, `runcloud`, `ippadmin`
### Cron (root)
```cron
0 3 * * * /usr/local/bin/lynis-scan.sh
```
### Custom Scripts
- `/usr/local/bin/lynis-scan.sh` (security scan, runs nightly at 3 AM)
---
## 7. 🔵 app1.itpropartner.com (n8n) — Workflow Automation
| Field | Value |
|-------|-------|
| **IP** | 87.99.144.163 |
| **Hostname** | n8n |
| **Plan** | CPX11 |
| **Specs** | 2 vCPU AMD EPYC-Rome · 1.9 GB RAM · 38 GB disk |
| **OS** | Ubuntu 24.04.4 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 14:35 |
| **Load** | 0.04 / 0.01 / 0.00 |
| **Disk** | 23% used (8.2G of 38G) |
### Docker Containers (2 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `n8n` | n8nio/n8n:latest | Up 3 days | 127.0.0.1:5678 |
| `n8n-postgres` | postgres:16-alpine | Up 3 days (healthy) | 5432 |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| n8nio/n8n | latest | 2.28 GB |
| docker.n8n.io/n8nio/n8n | latest | 2.28 GB |
| postgres | 16-alpine | 395 MB |
| hello-world | latest | 25.9 kB |
### Docker Compose
- `/opt/n8n/docker-compose.yml`
### Running Systemd Services (non-standard)
- `containerd.service`, `docker.service`
- `nginx.service` — **Nginx** (native, not RunCloud)
- `mariadb.service` — MariaDB 11.4.12
- `email@redacted`
- `fail2ban.service`, `firewalld.service`
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service` (installed but not running)
- `php81rc-fpm.service` through `php84rc-fpm.service` (installed but not running)
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80 | HTTP → Nginx |
| 443 | HTTPS → Nginx |
| 3306 | MariaDB |
| 5678 | n8n (localhost only) |
### Web Servers
- **Nginx** (native — proxying to n8n on 5678)
- Sites enabled: `app1.itpropartner.com`
### Databases
- **MariaDB** 11.4.12 (native, active)
- **Redis** (native, active)
- MySQL CLI
### Users
- `root`, `runcloud`, `ippadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 8. 🔵 docker — Docker Host / NPM / RustDesk
| Field | Value |
|-------|-------|
| **IP** | 178.156.168.35 |
| **Hostname** | docker |
| **Plan** | CPX11 |
| **Specs** | 2 vCPU AMD EPYC-Rome · 1.9 GB RAM · 38 GB disk |
| **OS** | Ubuntu 24.04.3 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:45 |
| **Load** | 0.25 / 0.47 / 0.38 |
| **Disk** | ⚠️ **73% used** (26G of 38G) |
### Docker Containers (7 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `nginx-proxy-manager` | jc21/nginx-proxy-manager:latest | Up 3 days | 80, 81, 443 |
| `uptime-kuma` | louislam/uptime-kuma:latest | Up 3 days (healthy) | 3001 |
| `hbbr` | rustdesk/rustdesk-server | Up 3 days | 21117, 21119 |
| `hbbs` | rustdesk/rustdesk-server | Up 3 days | 21114, 21116, 21118, 21116/udp |
| `strongswan` | dagavi/strongswan | Up 3 days | — |
| `runcloud_mariadb.*` | runcloud/mariadb:11.4 | Up 3 days (healthy) | — |
| `runcloud_php56rc.*` | runcloud/php56rc:latest | Up 3 days (healthy) | — |
| `runcloud_php81rc.*` | runcloud/php81rc:latest | Up 3 days (healthy) | — |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| jc21/nginx-proxy-manager | latest | 1.14 GB |
| runcloud/php56rc | latest | 521 MB |
| runcloud/php81rc | latest | 519 MB |
| louislam/uptime-kuma | latest | 448 MB |
| runcloud/mariadb | 11.4 | 419 MB |
| nginx | alpine | 53.7 MB |
| dagavi/strongswan | latest | 34.3 MB |
| rustdesk/rustdesk-server | latest | 12.8 MB |
### Docker Compose
- `/opt/litellm/docker-compose.yml`
- `/opt/openwebui/docker-compose.yml`
- `/opt/rustdesk/docker-compose.yml`
- `/opt/strongswan/docker-compose.yml`
### Running Systemd Services (non-standard)
- `containerd.service`, `docker.service`
- `fail2ban.service`
- `email@redacted`
- `supervisor.service`
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80, 81 | HTTP → NPM |
| 443 | HTTPS → NPM |
| 3001 | Uptime-Kuma |
| 3306 | MySQL (native) |
| 21114 | RustDesk (relay) |
| 21116 | RustDesk (TCP+UDP) |
| 21117 | RustDesk (relay) |
| 21118 | RustDesk |
| 21119 | RustDesk |
| 500, 4500 | StrongSwan IPsec |
| 2377, 7946 | Docker swarm |
### Databases
- **MySQL** (native, active on 3306 — likely RunCloud MariaDB)
### Web Servers
❌ Native — Nginx Proxy Manager runs in Docker
- RunCloud Docker containers for PHP
### Users
- `root`, `runcloud`, `ippadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 9. 🔵 fleettracker360 — GPS Fleet Tracking
| Field | Value |
|-------|-------|
| **IP** | 178.156.149.32 |
| **Hostname** | fleettrack360 |
| **Plan** | CPX11 |
| **Specs** | 2 vCPU AMD EPYC-Rome · 1.9 GB RAM · 38 GB disk |
| **OS** | Ubuntu 24.04.4 LTS (Noble Numbat) |
| **Kernel** | 6.8.0-134-generic |
| **Uptime** | 3 days 20:45 |
| **Load** | 0.00 / 0.00 / 0.00 |
| **Disk** | 25% used (8.9G of 38G) |
### Docker
❌ **No Docker installed**
### Running Systemd Services (non-standard)
- `traccar.service` — **Traccar GPS tracking** (Java)
- `apache2-rc.service` — Apache (RunCloud)
- `nginx-rc.service` — Nginx (RunCloud)
- `php81rc-fpm.service` — PHP-FPM 8.1
- `mariadb.service` — MariaDB 11.4.12
- `email@redacted`
- `fail2ban.service`, `firewalld.service`
- `runcloud-agent.service`
- `supervisor.service`
### Custom Systemd Services
- `apache2-rc.service`, `nginx-rc.service`
- `php81rc-fpm.service` through `php85rc-fpm.service`
- `runcloud-admin.service`, `runcloud-agent.service`, `runcloud-upgrade.service`
- `traccar.service`
### Listening Ports (very busy — Traccar uses many ports)
| Port | Service |
|------|---------|
| 22 | SSH |
| 25 | Postfix (SMTP) |
| 80 | HTTP → Nginx (RunCloud) |
| 443 | HTTPS → Nginx (RunCloud) |
| 81 | Apache (local proxy) |
| 3306 | MariaDB |
| 5001-5020 | Traccar (various tracking protocols) |
| 8082 | Traccar (web UI) |
| 34210 | RunCloud |
**UDP also: 5001-5032 — Traccar tracking protocols**
### Web Servers
- **nginx-rc** (Nginx w/ RunCloud — active frontend)
- **apache2-rc** (Apache w/ RunCloud — backend)
### Databases
- **MariaDB** 11.4.12 (native, active)
- **Redis** (native, active)
- MySQL CLI
### Users
- `root`, `runcloud`, `ippadmin`
### Cron
No crontab for root
### Custom Scripts
❌ None
---
## 10. 🔵 app1 (netcup) — Live Hermes (core)
| Field | Value |
|-------|-------|
| **IP** | 152.53.192.33 |
| **Hostname** | core |
| **Plan** | RS 2000 G12 (netcup) |
| **Specs** | 8 vCPU AMD EPYC 9645 · 15 GB RAM · 503 GB disk |
| **OS** | Debian 13 (trixie) |
| **Kernel** | 6.12.94+deb13-amd64 |
| **Uptime** | 3 days (from context) |
| **Disk** | 6% used (26G of 503G) |
### Docker Containers (7 running)
| Name | Image | Status | Ports |
|------|-------|--------|-------|
| `twenty-server-1` | twentycrm/twenty:latest | Up 34 hours (healthy) | 3001→3000 |
| `twenty-worker-1` | twentycrm/twenty:latest | Up 34 hours | — |
| `twenty-db-1` | postgres:16 | Up 34 hours (healthy) | 5432 |
| `twenty-redis-1` | redis | Up 34 hours (healthy) | 6379 |
| `docuseal` | docuseal/docuseal:latest | Up 2 days | 127.0.0.1:3000 |
| `searxng` | searxng/searxng:latest | Up 2 days | 127.0.0.1:8888 |
| `vaultwarden` | vaultwarden/server:1.33.2 | Up 3 days (healthy) | 8080→80 |
### Docker Images
| Repository | Tag | Size |
|------------|-----|------|
| twentycrm/twenty | latest | 1.12 GB |
| docuseal/docuseal | latest | 476 MB |
| postgres | 16 | 451 MB |
| searxng/searxng | latest | 258 MB |
| vaultwarden/server | 1.33.2 | 237 MB |
| redis | latest | 143 MB |
### Docker Compose
- `/opt/theharvester/docker-compose.yml`
### Running Systemd Services (non-standard)
| Service | Status | Purpose |
|---------|--------|---------|
| `hermes.service` | ✅ active | Hermes Agent |
| `hermes-browser.service` | ✅ active | Headless Chromium (CDP) |
| `caddy.service` | ✅ active | Caddy web server |
| `ollama.service` | ✅ active | Local LLM service |
| `mysql-tunnel.service` | ✅ active | AutoSSH tunnel → wphost02 MySQL |
| `shark-game.service` | ✅ active | Shark Attack Fantasy Game |
| `strongswan-starter.service` | ✅ active | IPsec VPN |
| `xl2tpd.service` | ✅ active | L2TP tunnel |
| `tailscaled.service` | ✅ active | Tailscale |
### Listening Ports (public)
| Port | Service |
|------|---------|
| 22 | SSH |
| 80 | Caddy (HTTP) |
| 443 | Caddy (HTTPS) |
| 8080 | Vaultwarden |
| 8081 | Hermes assistant |
| 8083 | Shark Game |
| 3001 | Twenty CRM |
| — | Internal services (localhost) |
| 3000 | Docuseal (localhost) |
| 8888 | SearXNG (localhost) |
| 9222 | Chrome DevTools (localhost) |
| 11434 | Ollama (localhost) |
| 33060 | MySQL tunnel (localhost) |
**UDP:** 500, 4500 (IPsec), 1701 (L2TP), 41641 (Tailscale), 51820 (WireGuard)
### Web Servers
- **Caddy** 2.11.4 (native — main reverse proxy with auto HTTPS)
### Caddy Config
- `sign.itpropartner.com` → proxy to Docuseal (127.0.0.1:3000), serves `/vehicles.json`
- `core.itpropartner.com` → serves static files, `/health`, `/capabilities`
### Databases
- MySQL CLI installed
- SQLite3 installed
- **No native MariaDB/PostgreSQL** — databases run in Docker
### Users
- `root`, `ippadmin`
### Cron (root)
```cron
* * * * * /root/.hermes/scripts/dre-mail-poller.py
0 * * * * /root/.hermes/scripts/boys-mail-monitor.py collect 2>&1 | logger -t boys-mail-collect
0 19 * * * /root/.hermes/scripts/boys-mail-monitor.py summary 2>&1 | logger -t boys-mail-summary
0 8 * * * /root/shark-game/scraper/run.sh
*/15 * * * * /root/.hermes/scripts/shark-draft-reminder.sh 2>&1 | logger -t shark-draft-reminder
```
### Custom Scripts
- `/root/explore_crm.py`, `explore_crm2.py`, `explore_crm3.py`
- `/root/fetch_zabbix_docs.py`
- `/root/kw_autocomplete.py`, `kw_paa_scrape.py`, `kw_research.py`
- `/root/parse_docs.py`, `parse_docs2.py`
- `/root/process_inbox.py`
- `/root/research_savannah.py`, `research_savannah2.py`
- `/root/test-feed.py`
- `/usr/local/bin/home-router-watchdog.sh`
---
## Cross-Cutting Summary
### Docker Hosts (5 of 10)
| Server | Containers | Notable |
|--------|-----------|---------|
| ai.itpropartner.com | 6 | LiteLLM, Open WebUI, Ollama, Qdrant |
| hudu.itpropartner.com | 5 | Hudu, SWAG, PostgreSQL, Redis |
| unms.forefrontwireless.com | 10 | UNMS, UCRM, RabbitMQ, PostgreSQL, Netflow |
| app1.itpropartner.com (n8n) | 2 | n8n, PostgreSQL |
| docker | 7+ | NPM, RustDesk, Uptime-Kuma, StrongSwan, RunCloud |
| app1 (netcup) | 7 | Twenty CRM, Vaultwarden, Docuseal, SearXNG |
### Non-Docker Hosts (4 of 10)
| Server | Native Apps |
|--------|------------|
| app1-bu | StrongSwan, L2TP, standby scripts |
| unifi | UniFi Controller, MongoDB, MariaDB |
| wphost02 | RunCloud (Nginx+Apache+PHP), MariaDB, WordPress |
| fleettracker360 | Traccar, RunCloud (Nginx+Apache+PHP), MariaDB |
### RunCloud Hosts (5 of 10)
hudu, unifi, unms, wphost02, fleettracker360 — all share RunCloud agent, custom Apache/Nginx/php*-fpm services
### Database Engines (across fleet)
| Engine | Servers |
|--------|---------|
| MariaDB/MySQL | hudu, unifi, unms, wphost02, n8n, fleettracker360, docker |
| PostgreSQL (Docker) | ai, hudu, unms, n8n, core (Twenty CRM) |
| MongoDB | unifi (UniFi Controller) |
| Redis | hudu, core, unms, wphost02, fleettracker360 |
| Qdrant | ai (vector DB) |
| SQLite | core |
### VPN / Networking
| Type | Server |
|------|--------|
| IPsec StrongSwan | app1-bu, docker, core |
| L2TP | app1-bu, core |
| Tailscale | core |
| WireGuard | core |
### Monitoring
| Tool | Server |
|------|--------|
| Uptime-Kuma | docker |
| Lynis security scan | wphost02 (cron) |
| Fail2ban | hudu, unifi, unms, wphost02, n8n, docker, fleettracker360 |
### Disk Warnings
- ⚠️ **ai.itpropartner.com**: 92% disk (199G of 226G) — critical
- ⚠️ **docker**: 73% disk (26G of 38G) — getting full
---
> **Audited:** 2026-07-09 via itpp-infra SSH key across all 10 servers.
references/hetzner-server-inventory.md
+148
View File
@@ -0,0 +1,148 @@
# Hetzner Server Inventory
**Generated:** July 9, 2026
**Purpose:** Migration planning, consolidation reference
---
## Summary: 10 Hetzner Servers
| Server | IP | Type | Uptime | Disk | RAM | Docker? | Key Service |
|--------|-----|------|--------|------|-----|---------|-------------|
| wphost02 | 5.161.62.38 | CPX21 | 7h | 67%/75G | 3.7G | No | RunCloud, Apex WP |
| unms | 5.161.225.131 | CPX21 | 4d 11h | 67%/75G | 3.7G | **Yes (9)** | UISP/UNMS, UCRM |
| unifi | 178.156.131.57 | CPX21 | 6h 42m | 14%/75G | 3.7G | No | UniFi Controller |
| hudu | 178.156.130.130 | CPX21 | 6h 42m | 23%/75G | 3.7G | **Yes (5)** | Hudu IT doc |
| fleettracker360 | 178.156.149.32 | CPX11 | 6h 46m | 25%/38G | 1.9G | No | Apache, MariaDB |
| docker | 178.156.168.35 | CPX11 | 4d 11h | **73%/38G** | 1.9G | **Yes (8)** | RunCloud, NPM, RustDesk |
| **ai** | 178.156.167.181 | **CPX41** | 4d 10h | **🔴 92%/226G** | **15G** | **Yes (6)** | **LiteLLM, Ollama, OpenWebUI** |
| app1-bu | 5.161.114.8 | CPX11 | 4d 4h | -/38G | 1.9G | No | **Warm standby** (Hermes) |
| app1 (n8n) | 87.99.144.163 | CPX11 | 6h 42m | 23%/38G | 1.9G | **Yes (2)** | n8n, n8n-postgres |
| tony-vps | 87.99.159.142 | CPX21 | 3d 6h | 11%/75G | 3.7G | No | Tony's Hermes |
---
## Server Details
### 1. wphost02 (5.161.62.38) — CPX21
- **Uptime:** 7h 35m (recently rebooted during SSH fix)
- **OS:** Debian 12/RunCloud managed
- **CPU:** 3 cores
- **RAM:** 3.7G total, 1.1G used
- **Disk:** 75G (67% used)
- **Docker:** No
- **Services:** nginx-rc (80/443), MariaDB (3306), Postfix (25), httpd (81), RunCloud agent (34210), node_exporter (9100), systemd-resolve (DNS)
- **Runs:** Apex Predators Track Experience (WordPress/WPForms), RunCloud panel
- **Notes:** Recently reconnected after SSH firewall fix. Mail relay configured.
### 2. unms.forefrontwireless.com (5.161.225.131) — CPX21
- **Uptime:** 4d 11h
- **CPU:** 3 cores
- **RAM:** 3.7G total, 1.8G used
- **Disk:** 75G (67% used)
- **Docker:** **Yes — 9 containers**
- **Containers:** unms-device-ws-1, unms-device-ws-2, unms-api, ucrm, unms-netflow, unms-siridb, unms-nginx, unms-rabbitmq, unms-postgres, unms-fluentd
- **Services:** Docker, MariaDB, containerd, fail2ban
- **Runs:** UISP/UNMS (WISP management), UCRM (customer management)
### 3. unifi (178.156.131.57) — CPX21
- **Uptime:** 6h 42m
- **CPU:** 3 cores
- **RAM:** 3.7G total, 1.2G used
- **Disk:** 75G (14% used)
- **Docker:** No
- **Services:** MariaDB, fail2ban, node_exporter
- **Runs:** UniFi Controller (Ubiquiti network management)
- **Notes:** Fresh install — only 6h uptime, 14% disk. Recently rebuilt?
### 4. hudu.itpropartner.com (178.156.130.130) — CPX21
- **Uptime:** 6h 42m
- **CPU:** 3 cores
- **RAM:** 3.7G total, 2.1G used
- **Disk:** 75G (23% used)
- **Docker:** **Yes — 5 containers**
- **Containers:** hudu2-app-1, hudu2-worker-1, hudu2-db-1, hudu2-redis-1, letsencrypt
- **Services:** Docker, MariaDB, firewalld
- **Runs:** Hudu (IT documentation platform)
### 5. fleettracker360 (178.156.149.32) — CPX11
- **Uptime:** 6h 46m
- **CPU:** 2 cores
- **RAM:** 1.9G total, 1.0G used
- **Disk:** 38G (25% used)
- **Docker:** No
- **Services:** Apache2, MariaDB, firewalld, fail2ban
- **Runs:** Fleet tracking application (BoxPilot Logistics?)
### 6. docker (178.156.168.35) — CPX11
- **Uptime:** 4d 11h
- **CPU:** 2 cores
- **RAM:** 1.9G total, 860M used
- **Disk:** 38G (**73% used** — tight)
- **Docker:** **Yes — 8 containers**
- **Containers:** RunCloud PHP (56/81), RunCloud MariaDB, nginx-proxy-manager, hbbr, hbbs (RustDesk), strongswan, uptime-kuma
- **Services:** Docker, containerd, fail2ban
- **Runs:** RustDesk self-hosted, Uptime Kuma monitoring, Nginx Proxy Manager, RunCloud staging
### 7. ai.itpropartner.com (178.156.167.181) — CPX41 ⭐
- **Uptime:** 4d 10h
- **CPU:** **8 cores**
- **RAM:** **15G** total, 3.9G used
- **Disk:** **226G (92% used — 🔴 CRITICAL)**
- **Docker:** **Yes — 6 containers**
- **Containers:** openwebui, litellm, litellm_postgres, caddy, qdrant, ollama
- **Services:** Docker, LiteLLM service, node_exporter, containerd
- **Runs:** **Key service: LiteLLM proxy for admin-ai.itpropartner.com**
- **Also runs:** Open WebUI, Ollama (local LLM), Qdrant (vector DB)
- **⚠️ 92% disk usage needs attention — 18G free on 226G drive**
### 8. app1-bu.itpropartner.com (5.161.114.8) — CPX11
- **Uptime:** 4d 4h
- **CPU:** 2 cores
- **RAM:** 1.9G total
- **Docker:** No
- **Services:** SSH, Tailscale, node_exporter, systemd-resolve
- **Ports:** 22 (SSH), 9999 (Hermes health?), 9100 (node_exporter), Tailscale
- **Runs:** Hermes **warm standby** — checks Core every 5 min, failover in 2 min. Has WG tunnel to home router.
### 9. app1.itpropartner.com / n8n (87.99.144.163) — CPX11
- **Uptime:** 6h 42m
- **CPU:** 2 cores
- **RAM:** 1.9G total, 853M used
- **Disk:** 38G (23% used)
- **Docker:** **Yes — 2 containers**
- **Containers:** n8n, n8n-postgres
- **Services:** Docker, MariaDB, firewalld, fail2ban
- **Runs:** n8n automation workflows
### 10. tony-vps (87.99.159.142) — CPX21
- **Uptime:** 3d 6h
- **CPU:** 3 cores
- **RAM:** 3.7G total, 1.1G used
- **Disk:** 75G (11% used)
- **Docker:** No
- **Services:** llama-server (Llama 3.2 1B fallback), qemu-guest-agent
- **Cron:** hermes-heartbeat.sh (every minute)
- **Runs:** Tony's standalone Hermes, local LLM fallback
- **Notes:** Minimal services — dedicated to Tony's Hermes agent
---
## Key Findings
### 🟢 Ready for migration to netcup:
- unms (UISP/UNMS) — Docker stack, straightforward
- n8n — Simple Docker stack
- docker box (RunCloud staging, NPM, RustDesk, Uptime Kuma)
- fleettracker360 — Apache+MariaDB, simple LAMP app
### 🟡 Needs care:
- **hudu** — Docker stack but sensitive data migration
- **unifi** — Recently rebuilt, needs verification
- **tony-vps** — Tony's separate Hermes, coordinate with him
### 🔴 Critical attention:
- **ai.itpropartner.com** — **92% disk full** (18G left on 226G). Runs LiteLLM, Ollama, OpenWebUI. **Primary model serving for all Hermes instances.** Cannot reboot without risk.
- **Disk cleanup needed:** 199G of 226G used. Docker images/logs likely the culprit.
### 🔵 Warm standby:
- **app1-bu** — Hermes clone, WG tunnel active, ready for failover
references/immich-mcp-reference.md
+58
View File
@@ -0,0 +1,58 @@
# Immich MCP Server — ITPP Reference
## Built: 2026-07-15
## Location
- Server: `/root/docker/immich-mcp/server.py`
- Service: `/tmp/immich-mcp.service` (pending install)
- Port: 8904
- Venv: `/opt/ops-portal/venv` (shared, contains fastmcp)
- Tools: 6 (immich_search, immich_asset_metadata, immich_map, immich_albums, immich_stats, immich_duplicates)
## Prerequisites (before deployment)
- [ ] `IMMICH_BASE_URL` in `/root/.hermes/.env` — URL of Immich server
- [ ] `IMMICH_API_KEY` in `/root/.hermes/.env` — generated in Immich → Account Settings → API Keys
## Deployment steps
```bash
cp /tmp/immich-mcp.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now immich-mcp
# Wire into Hermes
hermes config set mcp_servers.immich.url 'http://127.0.0.1:8904/mcp'
hermes config set mcp_servers.immich.enabled 'true'
hermes mcp test immich
```
## Immich API reference
- Auth: `x-api-key` header
- Base: `{IMMICH_BASE_URL}/api`
- Key endpoints used:
- `POST /search/metadata` — search by EXIF, GPS, date, camera, city, country
- `GET /assets/{id}` — full asset info
- `GET /assets/{id}/metadata` — EXIF metadata
- `GET /assets/{id}/thumbnail` — thumbnail image
- `GET /albums` — album list
- `GET /assets/statistics` — server stats
- `GET /duplicates` — duplicate detection
## S3 storage notes (Immich server side)
- Immich stores everything locally by default (`UPLOAD_LOCATION`)
- To use S3: mount bucket with `s3fs-fuse` at the upload location
- Pros: unlimited storage, no local disk limits
- Cons: FUSE latency, slow thumbnail/transcode, DB stays local
- Better approach: local SSD + nightly backup to S3 via cron
```bash
# Backup Immich assets to S3
aws s3 sync /path/to/immich/library s3://itpropartner-backups/immich/library/ --delete
aws s3 sync /path/to/immich/upload s3://itpropartner-backups/immich/upload/ --delete
# DB dump
pg_dump immich | aws s3 cp - s3://itpropartner-backups/immich/db/daily.sql.gz
```
- Backup cadence: daily (photos don't change often enough for 15-min sync)
## Backup recommendations
- DB + library assets: daily, S3
- Full backup: weekly, separate bucket
- Test restore: monthly (spin up test Immich container, point at restored DB + library)
references/incident-response-plan.md
+68
View File
@@ -0,0 +1,68 @@
# Incident Response Plan
## 1. Incident Severity Levels
* **P0 (Critical):** Production down, data loss, security breach.
* **Response Time:** Within 15 minutes
* **P1 (High):** Degraded service, single server down.
* **Response Time:** Within 1 hour
* **P2 (Medium):** Non-critical service issue.
* **Response Time:** Within 4 hours
* **P3 (Low):** Cosmetic, minor issue.
* **Response Time:** Within 24 hours
## 2. Alert Sources
Alerts are aggregated from the following monitoring systems:
* **Hermes `service-health-check.sh`:** Runs every 5 minutes, alerts routed to Telegram.
* **Hermes Standby Watchdog:** Monitors primary system health, alerts to Telegram and email upon failover.
* **Uptime Kuma (Docker Box):** External endpoint and service monitoring.
* **VPS Threshold Checks:** Alerts on CPU, Memory, Disk, and Network utilization limits.
* **Backup Audit Watchdog:** Daily verification of backup integrity and completion.
## 3. Response Team
* **CEO:** Germaine — Final decision authority, executive communications.
* **Lead Sysadmin:** Sho'Nuff — First responder, triage, root cause analysis, and resolution.
* **Operations Manager:** Anita — Internal and external communications, client notification.
* **Developer:** Tony — Application-level troubleshooting and code-level fixes.
## 4. Incident Response Steps
1. **Detect:** Alert received via Telegram, email, or client report.
2. **Triage:** Determine severity level (P0-P3), impact scope, and assign an Incident Owner.
3. **Contain:** Stop the bleeding. Immediate actions may include failing over to standby, blocking malicious IPs, or stopping affected services to prevent further damage or data loss.
4. **Investigate:** Identify the root cause, determine affected systems, and assess any data exposure.
5. **Resolve:** Apply the necessary fix and verify that the issue is mitigated.
6. **Recover:** Restore systems from backup if needed, and comprehensively verify all services are operating normally.
7. **Post-Mortem:** Document the root cause, establish a timeline of events, detail the applied fix, and define prevention measures.
## 5. Communication Plan
* **Internal Communication:**
* **P0/P1:** Immediate communication via dedicated Telegram incident channel.
* **P2/P3:** Communication via internal ticketing system and email.
* **Client Notification:** Notify affected clients within **1 hour** of a confirmed outage.
* **Update Frequency (During Active Incident):**
* **P0:** Updates every 30 minutes.
* **P1:** Updates every 2 hours.
## 6. Escalation Procedures
* **Initial Escalation:** If the first responder (Lead Sysadmin) does not acknowledge a P0/P1 alert within **15 minutes**, the alert automatically escalates to the **CEO**.
* **Vendor Escalation:** If the issue remains unresolved after **2 hours** of active investigation, escalate to external vendor support.
## 7. Key Contacts
* **netcup Support:** [Refer to `.env` for current contact details / PIN]
* **Hetzner Support:** [via Cloud Console / Support Ticket]
* **Wasabi Support:** [via Wasabi Management Console]
* **Cloudflare Support:** [via Cloudflare Dashboard]
* **MXroute Support:** [via MXroute Billing Portal / Support Desk]
## 8. Post-Incident Actions
* **Log Update:** Update the Disaster Recovery (DR) issue log with incident details.
* **Runbook/Skill Update:** If a new procedure, workaround, or troubleshooting step was discovered, document it and add it to the Hermes skills/runbooks.
* **Follow-up:** Schedule and track follow-up tasks to permanently fix the root cause and prevent recurrence.
references/ip-dns-changes.md
+92
View File
@@ -0,0 +1,92 @@
# IT Pro Partner — Infrastructure Rebalance: IP & DNS Changes
> **Last Updated:** July 12, 2026
> **Maintainer:** Sho'Nuff
## Server Roles
| Server | Current | New Role |
|---|---|---|
| **Core** (152.53.192.33 — netcup RS 2000) | Hermes, Caddy, Ops Portal, Super Search, SearXNG, Grafana, cron, LiteLLM | **Hermes + LiteLLM + Ollama (lightweight)** |
| **app1** (152.53.36.131 — netcup RS 4000) | n8n + Postgres, Ollama, Caddy (basic) | **Service Hub** — Caddy, Ops Portal, OSINT API, Super Search, SearXNG, Grafana, cron, n8n |
| **app2** (152.53.39.202 — netcup RS 4000) | UISP, UniFi, Traccar | Stays same |
| **app3** (152.53.241.111 — netcup RS 4000) | WordPress target | Stays same |
| **core-bu** (5.161.114.8 — Hetzner) | **app1-bu** → rename | Warm standby for Core (upgrade pending) |
---
## DNS Record Changes
### 🔄 Currently Live on SiteGround — Needs Master Action
| Domain | Old IP (Hetzner/SiteGround) | New IP (netcup) | Status |
|---|---|---|---|
| `unms.forefrontwireless.com` | **5.161.225.131** (old UNMS box) | **152.53.39.202** (app2) | ✅ Resolved to app2 (updated by Master) |
| `unifi.itpropartner.com` | **143.198.185.17** (old DigitalOcean) | **152.53.39.202** (app2) | ✅ Resolved to app2 (updated by Master) |
### 🔄 Currently on Master's DNS — Needs Action
| Domain | Old IP | New IP | Status |
|---|---|---|---|
| `hudu.itpropartner.com` | **178.156.130.130** (Hetzner CPX21) | **TBD** | ❌ Ready to migrate |
### 🔄 Currently on Cloudflare — Sho'Nuff Handles
| Domain | Old IP (Hetzner) | New IP (netcup) | Status |
|---|---|---|---|
| `fleettracker360.itpropartner.com` | **178.156.149.32** (Hetzner CPX11) | **TBD** | ❌ Ready to migrate |
### 🔄 Changes as Part of Core Rebalance
| Domain | Current → Points To | New → Points To | Change |
|---|---|---|---|
| `ops.itpropartner.com` | **152.53.192.33** (Core) | **152.53.36.131** (app1) | DNS update |
| `core.itpropartner.com` | **152.53.192.33** (Core) | **152.53.36.131** (app1) | DNS update |
| `sign.itpropartner.com` | **152.53.192.33** (Core) | **152.53.36.131** (app1) | DNS update |
| `admin-ai.itpropartner.com` | **178.156.167.181** (Hetzner old-ai) | **152.53.192.33** (Core) | DNS update |
| `app1.itpropartner.com` | **87.99.144.163** (old Hetzner app1) | **152.53.36.131** (netcup app1) | Already done? |
### ✅ Already Updated (No Change Needed)
| Domain | IP | Where |
|---|---|---|
| `app2.itpropartner.com` | 152.53.39.202 | netcup app2 |
| `app3.itpropartner.com` | 152.53.241.111 | netcup app3 |
| `crm.debtrecoveryexperts.com` | 152.53.192.33 | Core (Stays) |
| `portal.debtrecoveryexperts.com` | Cloudflare | Stays |
| `shark.iamgmb.com` | 152.53.192.33 | Core (Stays) |
| `tony.iamgmb.com` | 87.99.159.142 | Tony's VPS (Stays) |
---
## Servers to Cancel (After Migration Complete)
| Server | Current IP | Spec | Cost/mo | Cancel After |
|---|---|---|---|---|
| **old-ai** | 178.156.167.181 | CPX41 (4C/16G/226G) | ~$30 | LiteLLM moved to Core |
| **old app1** | 87.99.144.163 | CPX11 (2C/4G) | ~$10 | Confirmed deprecated |
| **docker box** | 178.156.168.35 | CPX11 (2C/4G) | ~$10 | Docker services migrated |
| **unms** | 5.161.225.131 | CPX21 (3C/8G) | ~$18 | DNS updated to app2 |
| **unifi** | **143.198.185.17** | CPX21 (3C/8G) | ~$18 | DNS updated to app2 |
| **hudu** | 178.156.130.130 | CPX21 (3C/8G) | ~$18 | Migrated to netcup |
| **fleettracker** | 178.156.149.32 | CPX11 (2C/4G) | ~$10 | Migrated to netcup |
| **wphost02** | 5.161.62.38 | CPX21 (3C/8G) | ~$18 | WP migrated to app3 |
## Servers to Keep
| Server | IP | Spec | Cost/mo | Role |
|---|---|---|---|---|
| **core-bu** (was app1-bu) | 5.161.114.8 | CPX11→**CPX31** (4C/16G) | ~$22 (upgrade) | Warm standby |
| **tony-vps** | 87.99.159.142 | CPX21 (3C/8G) | ~$18 | Tony's Hermes |
---
## Migration Priority Order
1. 🔴 **UniFi DNS** — SiteGround, Master action needed
2. 🔴 **UNMS DNS** — SiteGround, Master action needed
3. 🟡 **Core Rebalance** — Move services Core→app1 (detailed in core-rebalance-plan.md)
4. 🟡 **LiteLLM** — Move Hetzner→Core
5. 🟢 **Hudu Migration** — After target server decided, Master does DNS
6. 🟢 **Fleettracker Migration** — After target server decided, Sho'Nuff does Cloudflare DNS
7. 🟢 **Cancel dead servers** — After all migrations confirmed working
8. 🟢 **Upgrade core-bu** — CPX11→CPX31
references/itpp-recovery-manual.md
+1349
View File
File diff suppressed because it is too large Load Diff
references/master-apps-services.md
+340
View File
@@ -0,0 +1,340 @@
# IT Pro Partner — Master Apps & Services Inventory
> **Last Updated:** July 11, 2026
> **Maintainer:** Sho'Nuff
> **Purpose:** Complete registry of every installed application, subscription, API key, FOSS tool, and the server/service it lives on.
---
## Infrastructure Overview
| Server | IP | Specs | Provider | Role |
|---|---|---|---|---|
| **Core** | 152.53.192.33 | RS 2000 (8C/15G/512GB) | netcup | Ops hub |
| **app1** | 152.53.36.131 | RS 4000 (12C/32G/1TB) | netcup | AI/ML stack |
| **app2** | 152.53.39.202 | RS 4000 (12C/32G/1TB) | netcup | UISP/Infrastructure |
| **app3** | 152.53.241.111 | RS 4000 (12C/32G/1TB) | netcup | Web apps (spare) |
| **app1-bu** | 5.161.114.8 | CPX11 (2C/2G/40G) | Hetzner | Warm standby |
| **old-ai** | 178.156.167.181 | CPX41 (226G disk) | Hetzner | **Decommissioning** — admin-ai pending migration to app1 |
| **wphost02** | 5.161.62.38 | — | Hetzner | WordPress host (migrating to app3) |
| **Tony Hermes** | 87.99.159.142 | — | — | Anthony's personal Hermes |
**SSH Key:** `itpp-infra` (deployed to all servers, root SSH key-only)
**Base OS:** Debian 13 (all netcup boxes), Debian 12 (Hetzner boxes)
### Active Projects
| Project | Status | Server | Description |
|---|---|---|---|
| **Super Search** | ✅ Live v1.1.0 | Core | Multi-provider web search (SearXNG → Exa → Firecrawl) + extraction via FastMCP |
| **DRE / Debt Recovery Experts** | Active | Core + Cloudflare | Collections platform, CRM, approval workflows, letter templates |
| **Apex Track Experience** | Active | wphost02 (→ app3) | NASA Top Speed Event, WPForms, SMTP, PayPal |
| **Shark Attack Fantasy League** | Active | Core | shark.iamgmb.com, Discord bridge, changelog |
| **VoIPSimplicity** | Planning | — | RingLogix CPaaS, Yealink rentals |
| **Ops Portal** | Active | Core | ops.itpropartner.com, admin dashboards, customer portal |
| **admin-ai Migration** | Phase 0 Pending | old-ai → app1 | Move LiteLLM from Hetzner to netcup |
---
## Server-by-Server Inventory
### Core (netcup RS 2000 — 152.53.192.33)
#### System Services
| Service | Type | Purpose |
|---|---|---|
| **Hermes Agent** | systemd | AI assistant gateway (Telegram + cron) |
| **Caddy** | systemd | Reverse proxy (port 80/443) with Let's Encrypt |
| **Ops Portal** | systemd (uvicorn) | Internal admin dashboard at ops.itpropartner.com |
| **OSINT API** | systemd (uvicorn) | OSINT search API on port 8100 |
| **Hermes Assistant (PWA)** | systemd | PWA backend on port 8082 |
| **Shark Game** | systemd | Shark Attack Fantasy League backend on port 8083 |
| **Ollama** | systemd | Local LLM fallback (llama3.2:3b) |
| **Tailscale** | systemd | Mesh VPN for infra access |
| **StrongSwan + xl2tpd** | systemd | L2TP/IPsec VPN server |
| **WireGuard** | systemd | Tunnel to home router (10.77.0.0/24) |
| **MySQL Tunnel** | systemd | SSH tunnel to wphost02 MySQL (port 33060) |
| **Redis** | systemd | Cache layer |
| **Prometheus Node Exporter** | systemd | System metrics collection |
| **Hermes Browser** | systemd | Chromium headless for browser automation (9222) |
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **Komodo Core** | ghcr.io/moghtech/komodo-core:2 | 9120 | Docker management UI |
| **Komodo Mongo** | mongo | — | Komodo database |
| **Prometheus** | prom/prometheus | 9090 (loopback) | Metrics aggregation |
| **Grafana** | grafana/grafana | 3002 (loopback) | Monitoring dashboards |
| **Twenty CRM** | twentycrm/twenty:latest | 3001 | DRE CRM (crm.debtrecoveryexperts.com) |
| **Twenty Worker** | twentycrm/twenty:latest | — | Twenty async worker |
| **Twenty DB** | postgres:16 | — | Twenty PostgreSQL |
| **Twenty Redis** | redis | — | Twenty cache |
| **DocuSeal** | docuseal/docuseal:latest | 3000 (loopback) | Document signing |
| **SearXNG** | searxng/searxng:latest | 8888 (loopback) | Private search engine |
| **Vaultwarden** | vaultwarden/server:1.33.2 | 8080 | Password manager (tailnet only) |
#### Caddy Sites
| Domain | Backend | Purpose |
|---|---|---|
| sign.itpropartner.com | → :3000 (DocuSeal) | Document signing |
| core.itpropartner.com | static files | JSON data, capabilities, portal |
| crm.debtrecoveryexperts.com | → :3001 (Twenty) | DRE customer CRM |
| portal.debtrecoveryexperts.com | static | DRE customer portal |
| internal.debtrecoveryexperts.com | static | DRE internal dashboard |
| app.itpropartner.com | static + → :8082 | Hermes Assistant PWA |
| shark.itpropartner.com | static + → :8083 | Shark game |
| shark.iamgmb.com | static + → :8083 | Shark game alias |
| ops.itpropartner.com | → :8090 + static | Ops admin portal |
#### Cron Jobs (Hermes-managed, 22 total)
| Job | Schedule | Purpose |
|---|---|---|
| hermes-live-sync | Every 15m | S3 backup sync |
| service-health-check | Every 5m | Service monitoring |
| Home-Router-Watchdog | Every 5m | CCR watchdog |
| vps-threshold-check | Every 15m | Disk/mem alerts |
| hermes-memory-consolidate | Every 10m | Memory pruning |
| IMAP-email-triage | Every 60m | Email classification |
| IMAP-triage-watchdog | Every 10m | IMAP health check |
| shonuff-inbox-agent | Every 15m | Inbox monitoring |
| daily-tech-digest | Daily 11 AM | Tech news summary |
| exotic-vehicle-scout | Daily 9 AM | New car alerts |
| shark-scraper | Every 60m | Shark sighting data |
| dre-mail-poller | Every 1m | DRE inbox polling |
| dre-pending-approval-reminder | Hourly | DRE approval reminders |
| boxpilot-triage | Every 10m | BoxPilot email triage |
| bounce-check | Every 60m | Email bounce detection |
| apex-mail-watchdog | Every 5m | Apex SMTP health |
| ops-data-collector | Every 5m | Ops portal data |
| lynis-scan | Daily 3 AM | Security audit |
| firecrawl-usage-check | Daily 9 AM | Web search credits |
| model-usage-check | Daily 8 AM | LLM availability |
| hetzner-weekly-snapshots | Weekly Mon 5 AM | Server snapshots |
| home-router-daily-backup | Daily 6 AM | CCR config backup |
---
### app1 (netcup RS 4000 — 152.53.36.131) — AI/ML Stack
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **Ollama** | ollama/ollama:latest | 11434 (loopback) | Local LLM inference |
| **Open WebUI** | ghcr.io/open-webui/open-webui:latest | 3000 (loopback) | LLM chat UI (ai.itpropartner.com) |
| **n8n** | n8nio/n8n:latest | 5678 (loopback) | Workflow automation (n8n.itpropartner.com) |
| **n8n-postgres** | postgres:16-alpine | — | n8n database |
#### Caddy Sites
| Domain | Backend | Purpose |
|---|---|---|
| ai.itpropartner.com | → :3000 (Open WebUI) | LLM chat interface |
| n8n.itpropartner.com | → :5678 (n8n) | Automation workflows |
#### System Services
- Caddy, Docker, fail2ban, unattended-upgrades
#### Plans
- LiteLLM + Postgres (migrating from old-ai)
- Designated admin-ai.itpropartner.com target
---
### app2 (netcup RS 4000 — 152.53.39.202) — Infrastructure
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **UNMS API** | ubnt/unms:latest | — | UISP core API |
| **UCRM** | ubnt/unms-crm:4.5.33 | 80/443 | Customer CRM |
| **UNMS Nginx** | ubnt/unms-nginx:latest | 80/443/8089 | UISP web proxy |
| **UNMS Postgres** | ubnt/unms-postgres:latest | 5432 | UISP database |
| **UNMS SiriDB** | ubnt/unms-siridb:latest | — | UISP time-series DB |
| **UNMS RabbitMQ** | rabbitmq:3.7.28-alpine | 5672 | UISP message queue |
| **UNMS Netflow** | ubnt/unms-netflow:latest | 2055/udp | Network flow data |
| **UNMS Fluentd** | ubnt/unms-fluentd:latest | 24224 | Log aggregation |
#### System Services
Docker, fail2ban, unattended-upgrades
---
### app3 (netcup RS 4000 — 152.53.241.111) — Web Apps
**Status:** Fresh server, Docker installed, no containers yet. Designated for WordPress migrations from wphost02/SiteGround.
---
### app1-bu (Hetzner CPX11 — 5.161.114.8) — Warm Standby
#### System Services
| Service | Purpose |
|---|---|
| **Hermes** | Gateway dormant — auto-activates if Core fails (5 min watchdog) |
| **Ollama** | Local LLM fallback |
| **StrongSwan** | L2TP/IPsec VPN |
| **Tailscale** | Mesh VPN |
| **Node Exporter** | Metrics |
| **apex-status-receiver** | Apex endpoint |
---
### wphost02 (Hetzner — 5.161.62.38) — WordPress
| Site | Purpose |
|---|---|
| apextrackexperience.com | Apex Track Experience (WordPress + WooCommerce + WPForms) |
| apex-status | Apex status receiver endpoint |
**MySQL:** WordPress DB at `apextrackexperience_1781549652` (via Core tunnel on port 33060)
---
### old-ai (Hetzner CPX41 — 178.156.167.181)
**Status: Decommissioning.** All services migrating to app1.
| Container | Image | Purpose |
|---|---|---|
| LiteLLM | ghcr.io/berriai/litellm:v1.84.0 | LLM proxy (admin-ai) |
| LiteLLM Postgres | postgres:16 | Model configs, API keys |
| Caddy | caddy:latest | TLS termination |
| Qdrant | qdrant/qdrant:latest | Vector DB |
| Ollama | ollama/ollama:latest | Local models |
---
## Subscriptions & API Keys
### Active Subscriptions
| Service | Plan | Cost | Billing | Purpose |
|---|---|---|---|---|
| **netcup** | RS 2000 + 4× RS 4000 | ~$156/mo | Monthly | All servers |
| **Hetzner** | CPX11 + CPX41 + misc | ~$50/mo | Monthly | app1-bu, old-ai (decomm), wphost02, old boxes (decomm) |
| **MXroute** | Email hosting | ~$15/yr? | Annual | email@redacted, DRE email, boys' email |
| **SiteGround** | WordPress hosting | ~$10/mo | Monthly | Apex Track Experience (migrating) |
| **Cloudflare** | Free (DNS + CDN + Access) | Free | — | All domains |
| **OpenRouter** | Pay-as-you-go | Variable | Prepaid | Model API fallback |
| **Firecrawl** | Pay-as-you-go | Variable | — | Web scraping for AI |
| **ScrapingAnt** (or similar) | Pay-as-you-go | Variable | — | Web extraction |
| **SyncroMSP** | RMM | ~$2/mo per endpoint | Monthly | 26 customers, ITPP endpoints |
| **Wasabi S3** | Hot cloud storage | ~$7/TB/mo | Monthly | Hermes backups, router configs |
| **Nous Research** | Hermes Agent license | Usage-based | Prepaid | Hermes platform |
### API Keys & Tokens
| Key | Location | Service | Status |
|---|---|---|---|
| FIRECRAWL_API_KEY | ~/.hermes/.env | Web scraping | ✅ Active |
| CLOUDFLARE_API_TOKEN | ~/.hermes/.env | DNS management | ✅ Active |
| NETCUP_API_KEY | ~/.hermes/.env | Server provisioning | ✅ Active |
| NETCUP_PASSWORD | ~/.hermes/.env | SCP API auth | ✅ Active |
| SYNCROMSP_API_TOKEN | ~/.hermes/.env | RMM platform | ✅ Active |
| TELEGRAM_BOT_TOKEN | ~/.hermes/.env | Telegram bot | ✅ Active |
| LITELLM_MASTER_KEY | ~/.hermes/config.yaml + /opt/ai/.env | admin-ai proxy | ✅ Active |
| LITELLM_SALT_KEY | /opt/ai/.env | Admin-ai encryption | ✅ Active |
| OpenRouter API Key | ~/.hermes/config.yaml | Model API + credits | ✅ Active |
| Hermes Nous Portal Key | — | Hermes updates | ❌ Not logged in |
| SERVER_PASSWORDS | ~/.hermes/.env | Server root passwords | ✅ Active |
| DeepSeek API Key | Litellm Postgres (encrypted) | DeepSeek LLM API | ✅ Active |
| Anthropic API Key | LiteLLM Postgres (encrypted) | Claude models | ✅ Active |
| Google/Gemini API Key | LiteLLM Postgres (encrypted) | Gemini vision + text | ✅ Active |
| OpenAI API Key | LiteLLM Postgres (encrypted) | GPT models | ✅ Active |
| Groq API Key | LiteLLM Postgres (encrypted) | Fast inference | ✅ Active |
| xAI API Key | LiteLLM Postgres (encrypted) | Grok models | ✅ Active |
---
## FOSS Software Stack
### Core Infrastructure
| Software | License | Server | Purpose |
|---|---|---|---|
| Debian 13 | GPL | All | Server OS |
| Docker + Compose | Apache 2.0 | All | Container runtime |
| Caddy | Apache 2.0 | Core, app1 | Reverse proxy + TLS |
| Tailscale | BSD | Core, app1-bu | Mesh VPN |
| WireGuard | GPL | Core | Site-to-site VPN |
| StrongSwan | GPL | Core, app1-bu | L2TP/IPsec VPN |
| Prometheus | Apache 2.0 | Core | Metrics |
| Grafana | AGPL | Core | Dashboards |
### Applications
| Software | License | Server | Purpose |
|---|---|---|---|
| **Hermes Agent** | Proprietary | Core, app1-bu | AI assistant |
| **Ollama** | MIT | Core, app1, app1-bu | Local LLM |
| **Open WebUI** | MIT | app1 | LLM chat UI |
| **n8n** | Sustainable Use License | app1 | Workflow automation |
| **Twenty CRM** | AGPL | Core | Customer CRM |
| **DocuSeal** | AGPL | Core | Document signing |
| **SearXNG** | AGPL | Core | Private search |
| **Vaultwarden** | GPL | Core | Password manager |
| **Komodo** | GPL | Core | Docker management |
| **LiteLLM** | MIT | old-ai (→ app1) | LLM proxy |
| **Qdrant** | Apache 2.0 | old-ai | Vector DB |
| **UISP/UNMS** | Proprietary | app2 | ISP management |
| **UCRM** | Proprietary | app2 | Customer management |
| **Ollama Models** | Various | Core, app1, old-ai | LLM weights |
| **Redis** | BSD | Core | Cache |
| **PostgreSQL** | PostgreSQL | Core, app1, app2 | Databases |
| **WordPress** | GPL | wphost02 (→ app3) | Web content |
| **WooCommerce** | GPL | wphost02 | E-commerce |
| **WPForms** | GPL | wphost02 | Form builder |
| **BlogWatcher** | — | Core cron | RSS monitoring |
| **Lynis** | GPL | Core | Security auditing |
---
## Domains & DNS
| Domain | Registrar | DNS | Purpose |
|---|---|---|---|
| itpropartner.com | — | Cloudflare | Primary |
| germainebrown.com | — | MXroute | Email |
| debtrecoveryexperts.com | — | Cloudflare | DRE |
| apextrackexperience.com | — | SiteGround/Cloudflare | Apex |
| iamgmb.com | — | MXroute | Boys' email |
| voipsimplicity.com | — | Cloudflare | VoIPSimplicity |
| tailc2f3b0.ts.net | — | Tailscale | Tailnet DNS (internal) |
---
## Backup & DR
| Backup | Schedule | Location | Retention |
|---|---|---|---|
| Hermes full backup | Daily 5 AM | s3://hermes-vps-backups/hermes-full-backup/ | — |
| Hermes live state | Every 15 min | s3://hermes-vps-backups/live/ | Rolling |
| Router config (home) | Daily 6 AM | s3://mikrotik-ccr-backups/ | — |
| Root essentials | Daily | S3 | — |
| Hetzner snapshots | Weekly Mon | Hetzner Cloud | — |
**DR Standby:** app1-bu (Hetzner) boots and auto-restores from S3 if Core is down for 2+ minutes.
---
## Email Accounts
| Address | Provider | Server | Purpose |
|---|---|---|---|
| email@redacted | MXroute | mail.germainebrown.com | Germaine's primary |
| email@redacted | MXroute | mail.germainebrown.com | Sho'Nuff's outbox |
| email@redacted | SiteGround | c1113726.sgvps.net:2525/993 | Apex business |
| email@redacted | MXroute | — | DRE client-facing |
| email@redacted | MXroute | — | DRE debtor-facing |
| email@redacted | MXroute | — | Garrison Brown |
| email@redacted | MXroute | — | Greyson Brown |
references/memory-consolidation-proposal.md
+123
View File
@@ -0,0 +1,123 @@
# Hermes Memory Consolidation — Architecture Proposal
**Author:** Sho'Nuff
**Date:** July 9, 2026
**Status:** DRAFT — awaiting Network Services Team input
---
## 1. Current Problem
MEMORY.md is a flat list of freeform text entries separated by `§`.
The consolidation system (`hermes-consolidate.py`) prunes stale entries using:
- **A static PROTECT regex** listing keywords that prevent deletion
- **Stale-word detection** (`complete`, `done`, `resolved`, etc.)
- **A size safety valve** that drops oldest-entered entries when over 7,000 chars
**Failure observed on first run:** 3 critical entries were pruned because they
didn't match the PROTECT regex. The S3 backup preserved them, but without that
safety net they would have been permanently lost.
### Root Cause
The system has no way to **judge importance**. It treats all unprotected entries
equally and has no semantic understanding of what's durable vs ephemeral.
---
## 2. Proposed Solutions
### Option A: Priority Tagging (Recommended)
Add explicit priority metadata to each entry:
```
[P1] Core: netcup KVM, 8C/15G/512GB, Debian 13...
[P1] Germaine's #1 rule...
[P2] Apex on wphost02...
[P3] Shark game updates complete...
```
**Pruner logic:**
1. Keep all `[P1]` entries (never pruned)
2. Keep `[P2]` entries unless over 80% of limit
3. Prune `[P3]` entries first (completed tasks, transient state)
4. Never prune below `[P1]` + `[P2]` combined size
**Pros:** Simple, explicit, user can tag importance when writing
**Cons:** Requires tagging discipline at entry creation time
### Option B: Dual-Store Architecture
Split memory into two files:
```
memories/MEMORY.md — Durable facts (never auto-pruned)
memories/WORKING.md — Transient state (auto-pruned aggressively)
```
**Durable store (MEMORY.md):** identity, rules, credentials, infrastructure,
user preferences, team definitions. Only manually edited. No auto-prune.
**Working store (WORKING.md):** task status, queue items, completed work,
session notes, temporary context. Auto-pruned every 10 min.
**Pros:** No risk of losing durable facts. Working store can be more aggressive.
**Cons:** Two files to manage. Boundary between them requires judgement.
### Option C: Template-Based Structuring
Each entry follows a defined schema:
```yaml
type: infrastructure | rule | preference | task | personal | team
priority: p1 | p2 | p3
expires: 2026-08-01 | never
content: ...
```
**Pruner:**
- `expires: never` and `priority: p1` → never pruned
- `priority: p3` or past expiry → always pruned
- Others → size-based pruning
**Pros:** Surgical precision. No false positives.
**Cons:** Highest initial investment. More complex parsing. Schema changes
require migration.
---
## 3. Comparison
| Criteria | Current | Option A (Tags) | Option B (Dual) | Option C (Schema) |
|----------|---------|-----------------|------------------|-------------------|
| Setup effort | ✅ Done | 🟡 1 hour | 🟡 1 hour | 🔴 3+ hours |
| Risk of data loss | 🔴 HIGH | 🟡 Low | ✅ Minimal | ✅ Minimal |
| Maintenance burden | 🟡 Medium | 🟡 Low | ✅ Low | 🔴 Medium |
| User discipline needed | 🔴 None (but risky) | 🟡 Some | 🟡 Some | 🔴 High |
| Machine-readable | ❌ No | 🟡 Partial | 🟡 Partial | ✅ Yes |
| Migration complexity | — | 🟡 Low | 🟡 Low | 🔴 Medium |
---
## 4. Recommendation
**Option A (Priority Tagging) + apply to dual-store concept:**
- Keep one MEMORY.md for now
- Add `[P1]`, `[P2]`, `[P3]` tags
- Auto-tag entries where possible (infrastructure = P1, completed tasks = P3)
- The consolidation system protects P1 + P2, prunes P3 aggressively
- If the file still grows, split into durable/working stores in v2
---
## 5. DR Plan per Server — Note
This proposal covers Hermes memory only. Germaine has also requested a
comprehensive Disaster Recovery plan for **each server** in the ITPP
infrastructure. That is a separate deliverable currently being compiled
in `/root/.hermes/references/server-dr-plans.md`.
*Network Services Team review requested.*
references/model-price-research.md
+95
View File
@@ -0,0 +1,95 @@
# Model Recommendations — Best Price/Performance to Add to Admin-AI
**Research date:** July 9, 2026
**Authored by:** Research Team via Sho'Nuff
---
## Already Covered (Direct Through Admin-AI — No OpenRouter)
| Model | In/Out per 1M | Class |
|---|---|---|
| deepseek-chat (V3) | $0.14 / $0.28 | Budget generalist |
| deepseek-v4-pro | $0.435 / $0.87 | Strong reasoning |
| deepseek-v4-flash | $0.15 / $0.30 | Fast budget |
| deepseek-reasoner (R1) | $0.55 / $2.19 | Deep reasoning |
| deepseek-coder | $0.14 / $0.28 | Code specialist |
| gemini-flash-latest | $0.08–0.30 / $0.30–2.50 | Cheapest |
| gemini-pro-latest | $1.25 / $5.00 | Strong vision |
| claude-haiku-4-5 | $1.00 / $5.00 | Fast Anthropic |
| claude-sonnet-4-6 | $3.00 / $15.00 | Balanced Anthropic |
| claude-opus-4-8 | $5.00 / $25.00 | Best reasoning |
| GPT-5.5 | $5.00 / $30.00 | OpenAI flagship |
| grok-4.3 | ~$3.00+ | xAI reasoning |
## 🔥 Top Additions — Best Price/Performance
### Tier 1: Must Add (Cheap + Strong)
| Model | In/Out per 1M | Context | Provider | Why Add |
|---|---|---|---|---|
| **Qwen 3.5 27B** | ~$0.35 / $0.35 | 128K | Together / Groq / Fireworks | Best open-weight coding model. SWE-bench 77% — beats Claude Sonnet on code at 1/40th the price |
| **Qwen 3.6 Plus** | ~$0.80 / $0.80 | 128K | Alibaba Cloud | Stronger than 3.5, 120B class reasoning |
| **Mistral Small** | $0.10 / $0.30 | 128K | Mistral | European, GDPR-friendly. Cheapest general-purpose option |
| **Llama 4 (70B)** | ~$0.60 / $0.90 | 1M | Together / Groq | Open-weight. 1M context window at $0.60 input. Good generalist |
| **GPT-5.5 Mini** | $1.50 / $6.00 | 256K | OpenAI | Budget GPT — strong reasoning at 1/3 flagship price |
### Tier 2: Strong Value (Specific Use Cases)
| Model | In/Out per 1M | Context | Provider | Why Add |
|---|---|---|---|---|
| **Gemini 3.1 Flash-Lite** | $0.10 / $0.40 | 1M | Google | Even cheaper than Flash. 1M context. For bulk tasks |
| **Mistral Large** | $2.00 / $6.00 | 128K | Mistral | European Claude competitor. GDPR-compliant. Strong reasoning |
| **Grok 4 Mini** | $0.30 / $0.50 | 128K | xAI | xAI's budget model. Already have Grok 4.3, add the cheap one |
| **Llama 4 405B** | $3.50 / $3.50 | 128K | Together | Largest open model. For tasks needing maximum open-model power |
### Tier 3: Specialty Models
| Model | In/Out per 1M | Context | Provider | Why Add |
|---|---|---|---|---|
| **Qwen 3.6 Coder** | ~$0.35 / $0.35 | 128K | Alibaba | Code-focused version of Qwen's best |
| **Codestral** | ~$0.30 / $0.90 | 256K | Mistral | Mistral's code-specialized model |
| **MiniMax M2.5** | ~$0.50 / $2.00 | 256K | MiniMax | Chinese budget option, 1M context |
---
## Cost Comparison — Current vs Recommended
| Scenario | All DeepSeek (current) | With Qwen/Mistral mixed | Annual savings |
|---|---|---|---|
| 1M requests/month (avg 1000 in, 500 out tokens) | ~$360/mo | ~$90/mo | **$3,240/yr** |
| 10M requests/month | ~$3,600/mo | ~$900/mo | **$32,400/yr** |
Qwen 3.5 27B at $0.35/$0.35 handles 80% of coding tasks at 1/3 the cost of deepseek-v4-pro. Mistral Small at $0.10/$0.30 handles simple Q&A at less than half of deepseek-chat.
---
## How to Add Them
All of these are available through API gateways already supported by LiteLLM (admin-ai):
**Via Together AI** (single API key, multiple models):
- Qwen 3.5 27B: `together_ai/qwen/Qwen3.5-27B`
- Llama 4 70B: `together_ai/meta-llama/Llama-4-70B`
- Qwen 3.6 Coder: `together_ai/qwen/Qwen3.6-Coder`
**Via Groq** (fastest inference, free tier available):
- Qwen 3.5: `groq/qwen-3.5-27b`
- Llama 4: `groq/llama-4-70b`
- Mistral models: via Groq or direct
**Via Mistral API** (direct):
- Mistral Small: `mistral/mistral-small-latest`
- Mistral Large: `mistral/mistral-large-latest`
**Via xAI API** (likely already configured):
- Grok 4 Mini: `xai/grok-4-mini`
---
## Immediate Recommendation
1. **Get a Together AI API key** ($5 credit free) — gives Qwen 3.5 27B + Llama 4 + Mistral access through one account
2. **Add `together_ai/qwen/Qwen3.5-27B`** — single best price/performance model available anywhere right now
3. **Add `mistral/mistral-small-latest`** — $0.10/$0.30 is unbeatable for simple tasks
4. **Add the Together models to admin-ai's LiteLLM config** — one API key unlocks 5+ high-value models
references/network-diagram.md
+86
View File
@@ -0,0 +1,86 @@
# IT Pro Partner Infrastructure - Network Diagram
## Network Topology (ASCII)
```text
[ Users / Internet ]
| (HTTP/S)
v
+-------------------+
| Cloudflare |
| (CDN / DNS) |
+---------+---------+
|
v
+-------------------+
| Primary Host |
| (Core) |
| - Caddy Proxy |
| - WireGuard Hub |
| - SMTP Relay |
+--+------+------+--+
| | |
(WireGuard Port 51821) | | | (Tailscale Mesh)
+-----------------------+ | +------------------------+
v | v
[Home Router] | [Standby Host]
| (app1-bu)
v
+-------------------+-------------+-------------+-------------------+
| | | |
v v v v
[Host A] [Host B] [Host C] [Legacy Net]
(app1) (app2) (app3) (Hetzner)
AI Stack Net Mgmt Web Hosting - AI Server
- Docker Box
- UNMS
- UniFi
- wphost02 <-+
|
[Primary Host] ---(SSH Tunnel Port 33060 for MySQL)--------------------------+
```
## Server Networks & IP Addressing
### Internet Entry Points
* **Cloudflare CDN/DNS**: Proxies all public domains.
* **Caddy Reverse Proxy**: Hosted on Primary Host (Core), acts as the central ingress routing to internal services.
### Active Server Networks
* **Primary Host (Core)**: `152.53.192.33` — WireGuard hub (`10.77.0.1`), SMTP relay (port 2525), Caddy reverse proxy.
* **Host A (app1)**: `152.53.36.131` — AI stack (Ollama).
* **Host B (app2)**: `152.53.39.202` — Networking management.
* **Host C (app3)**: `152.53.241.111` — Web hosting.
* **Standby Host (app1-bu)**: `5.161.114.8` — Warm standby, connected via Tailscale to Core.
### Legacy Hetzner Network (Pending Migration/Decommission)
* **AI Server**: `178.156.167.181` — Migrating to app1.
* **Docker Box**: `178.156.168.35` — nginx-proxy-manager, Uptime Kuma.
* **wphost02**: `5.161.62.38` — Decommissioning (MySQL accessed via SSH tunnel from Core).
* **UNMS**: `5.161.225.131` — Migrating to app2.
* **UniFi**: `178.156.131.57` — Migrating to app2.
## Network Paths & VPN/Mesh Details
* **Ingress Path**: External Traffic -> Cloudflare (Ports 80/443) -> Primary Host (Core) -> Caddy Reverse Proxy -> Target Application/Service (Local or cross-host routing).
* **WireGuard Network**: Connects Primary Host/Core (`10.77.0.1`) and Home Router (`10.77.0.2`) on UDP port 51821.
* **Tailscale Mesh**: Secure mesh connection established between Primary Host (Core) and Standby Host (app1-bu) for warm standby synchronization and secure administrative access.
* **Site-to-Site SSH Tunnel**: Configured from Primary Host (Core) to legacy `wphost02`, forwarding local port 33060 to access the legacy MySQL database.
## Service Ports Table
| Port | Service/Protocol | Description / Routing |
| :--- | :--- | :--- |
| **22** | SSH | UFW-managed, key-only access across all hosts. |
| **80 / 443** | HTTP/HTTPS | Standard web traffic proxied via Caddy to backend services. |
| **2525** | SMTP Relay | Hosted on Primary Host (outbound email relay only). |
| **3001** | Uptime Kuma | Monitoring service on Legacy Docker Box. |
| **8089** | UNMS | Ubiquiti Network Management System (Legacy, migrating to app2). |
| **8443** | UniFi | UniFi Network Controller (Legacy, migrating to app2). |
| **11434** | Ollama | AI service running on Host A (app1). |
| **33060** | MySQL | Forwarded over SSH tunnel from Core to `wphost02`. |
| **51821** | WireGuard | Core-to-Home Router VPN tunnel. |
## DNS Zones
* All public domains are managed and proxied via **Cloudflare CDN/DNS**.
* Cloudflare acts as the primary defense layer and DNS resolution endpoint, obscuring internal host IP addresses for edge-routed traffic.
references/ops-portal-audit-2026-07-12.md
+44
View File
@@ -0,0 +1,44 @@
# Ops Portal Full Bug Audit Report
**Date:** July 12, 2026
**Scope:** All 11 pages at ops.itpropartner.com
**Auditor:** Claude Sonnet 4-6 (via delegation)
**Total bugs found:** 42 (14 Critical, 9 High, 12 Medium, 7 Low)
## Root Cause Summary
1. `/js/utils.js` returned 404 — file existed at `/var/www/ops/js/utils.js` but not at `/opt/ops-portal/static/js/` (the backend's JS directory)
2. `window.initNav` not exported from app.js — only exists at `Ops.initNav`. Every page calls `window.initNav()` which is undefined
3. `logoutBtn` DOM lookups happen before nav fetch completes on 3 pages
4. `window.fetchStatus` undefined on cost.html — hard crash on load
5. 6 pages have no auth guard at all
6. audit.html uses sessionStorage instead of localStorage for token
## Priority Fix List
### 🔴 P1 — Critical (fix first)
| Bug | Pages | Fix |
|-----|-------|-----|
| `utils.js` 404 | All 10 pages | ✅ Already fixed — copied to backend static dir |
| `window.initNav` undefined | All fetched-nav pages | Export `window.initNav = initNav` in app.js |
| `logoutBtn` null ref before nav | services, servers, backups | Move getElementById inside fetch callback |
| `window.fetchStatus` crash | cost.html | Replace with raw fetch() call |
| `window.escapeHtml` undefined | config.html | Use Ops.escapeHtml or add window alias |
| audit.html uses sessionStorage | audit.html | Change to localStorage |
| No auth guard | network, cron, config, logs, cost, fleet | Add login overlay + auth check |
### 🟠 P2 — High
| Bug | Fix |
|-----|-----|
| Login overlay hardcoded `show` class | Remove `show` from HTML |
| `<main>` misplaced in cron.html | Move `<main>` to wrap content |
| Grafana Tailscale IP breaks on mobile | Replace with hostname or remove |
### 🟡 P3 — Medium
| Bug | Fix |
|-----|-----|
| `summary-bar` CSS missing from ops.css | Add styles |
| fleet tracker missing viewport-fit=cover | Add to meta |
| Hardcoded username `germaine` in audit.html | Remove value attribute |
| COT midnight hour bug in fleet tracker | Use % 24 |
## Full Bug List (42 items)
See `/root/.hermes/cache/delegation/subagent-summary-0-20260712_081532_243138.txt` for complete details.
references/ops-portal-recommendations.md
+549
View File
@@ -0,0 +1,549 @@
# IT Pro Partner — Ops Portal Recommendations
> Generated: 2026-07-09
> Based on audit of current portal at `/var/www/ops/` and ops-status.json collection pipeline
---
## Table of Contents
1. [Current State Summary](#1-current-state-summary)
2. [Recommended Pages (New & Improved)](#2-recommended-pages)
3. [Actions Per Page — What Users Should Be Able to Do](#3-actions-per-page)
4. [Integrations Roadmap](#4-integrations-roadmap)
5. [Data Already Flowing](#5-data-already-flowing)
6. [RBAC Recommendations](#6-rbac-recommendations)
7. [Implementation Priorities](#7-implementation-priorities)
---
## 1. Current State Summary
### Existing Pages (all read-only, no action buttons)
| Page | URL | Status | What It Shows |
|------|-----|--------|---------------|
| Dashboard | `/` | ✅ Live | Health bar, cron jobs, services, S3 backups, API health, server inventory, versions |
| Servers | `/servers.html` | ✅ Live | netcup detail + Hetzner table (sortable), quick specs |
| Backups | `/backups.html` | ✅ Live | S3 bucket status grid, backup schedules, Wasabi info, manual actions (text only) |
| Network | `/network.html` | ✅ Live | DNS/Cloudflare zones, MikroTik (placeholder), Ubiquiti (placeholder) |
| Services | `/services.html` | ✅ Live | Systemd service status grid, API health, SyncroMSP integration summary |
| Cron Jobs | `/cron.html` | ✅ Live | Metrics overview, searchable job table with script viewer |
| Logs & Events | `/logs.html` | ✅ Live | API timeline, service events, cron errors, failure summary with filters |
| Config | `/config.html` | ✅ Live | Versions, config file viewer (Caddyfile, Hermes YAML), environment |
| Template | `/template.html` | ✅ Minimal | Empty page with nav injection |
### Key Gaps Identified
- **No action/control buttons anywhere** — everything is read-only
- **No customer management** — no client list, activations, lifecycle
- **No billing/finance** — Stripe, Xero, invoices, payments
- **No user/RBAC management** — no login, no roles
- **No audit log** — no record of who did what
- **No settings/integrations page** — API keys, integrations, branding
- **MikroTik routers** — placeholder only
- **Ubiquiti/UISP** — placeholder only
- **No backup restore buttons**
- **No service start/stop/restart**
- **No provisioning actions** — no way to spin up a server or create a customer
- **No DNS management actions** — cannot add/remove records
- **No support ticket creation**
---
## 2. Recommended Pages
### 2.1 Dashboard (Improve Existing)
**Current:** Health bar, cron jobs, S3 backups, API health, server inventory, versions.
**Improvements:**
- Add **role-aware widgets** — show different cards depending on who's logged in
- Add **quick-action toolbar** — "Provision new server", "New customer", "Create ticket"
- Add **alert/incident summary** — current failures, threshold breaches
- Add **recent activity feed** — last 10 actions taken in the portal
- Add **revenue/ billing snapshot** for owner role
- Add **weather/network status** at a glance
- Add **pinned services** — most-watched services at top
### 2.2 Server Management (Improve Existing)
**Current:** netcup detail, Hetzner table with sorting, quick specs.
**Improvements:**
- Add **provisioning actions** per server:
- Restart server (soft reboot via Hetzner API)
- Power on/off
- Create snapshot
- Open SSH console (WebSSH / terminal-in-browser)
- View resource graphs (CPU, RAM, disk over time)
- Add **threshold alerts per server** — show warning if disk/mem/CPU exceeds thresholds
- Add **server tags/groups** — tag servers by role (web, db, ai, monitoring)
- Add **migration status** — track Hetzner→netcup migration progress per server
- Show **backup status per server** (last Hetzner snapshot date)
- Show **Hudu link** for each server's documentation page
- Add **network interface stats** — bandwidth per interface
### 2.3 Customer Management (NEW)
**Purpose:** One place to see all ITPP customers and their lifecycle.
**Data sources:** SyncroMSP, Stripe, Hudu, DRE CRM
**Display:**
- Customer list/search table with: name, email, phone, plan/service tier, status (active/pending/suspended), billing status, last ticket date
- Customer detail page with:
- Contact info
- Active services/subscriptions
- Ticket history (from SyncroMSP)
- Billing history (from Stripe)
- Device inventory (from SyncroMSP)
- Notes / tags
- Linked documentation (Hudu link)
**Actions:**
- Create new customer
- Suspend/reactivate customer
- Assign service tier
- Create ticket on behalf of customer
- View full billing history
- Link to SyncroMSP, Hudu, Stripe
### 2.4 Network (Improve Existing)
**Current:** Cloudflare zones + placeholders for MikroTik and Ubiquiti.
**Improvements:**
**A) MikroTik Routers (unlock placeholder):**
- Router list with: hostname, model, RouterOS version, uptime, CPU/load, memory, interfaces
- Interface table: name, status (up/down), traffic (tx/rx), errors
- Connected clients / DHCP leases
- Firewall rules overview (count, active)
- Backup status (link to MikroTik backup on Wasabi)
**Actions:**
- Reboot router
- Run configuration backup
- View interface traffic graphs
- Enable/disable interface
- View log tail
**B) Ubiquiti / UISP (unlock placeholder):**
- UISP device list: towers, access points, CPEs
- Client counts per tower
- Bandwidth utilization graphs
- Device health (online/offline, signal strength)
- UniFi controller status, access point health
**Actions:**
- Provision new CPE
- Block/stick client
- View real-time RF metrics
- Trigger site survey
**C) DNS & Domains (add actions):**
Currently 13 Cloudflare zones already tracked:
boxpilotlogistics.com, broadbandtx.com, debtrecoveryexperts.com, fleettracker360.com, forefrontisp.com, germainebrown.com, iamgmb.com, itpropartner.com, katiewattsdesign.com, localisp.com, texasinternetaccess.com, vigilanttac.com, voipsimplicity.com
**Actions to add:**
- Add/edit/delete DNS records (A, CNAME, MX, TXT, etc.)
- Enable/disable proxy (Cloudflare orange cloud)
- View DNS record count per zone
- Trigger Cloudflare cache purge
- Show SSL/TLS status per domain
- Domain expiry dates (if registrar API connected)
### 2.5 Backups (Improve Existing)
**Current:** S3 bucket status grid, schedules, Wasabi info, text-only manual actions.
**Improvements:**
- Add **one-click restore** button per bucket (list snapshots, pick restore point)
- Add **trigger backup now** button per bucket
- Show **backup size** per bucket (not just status)
- Add **restore logs** — history of recent restores
- Add **backup retention policy** display (how long each bucket keeps data)
- Add **integrity check results** — last verification timestamp and status
- Add **restore to staging** — restore to a temp directory for verification
**Actions:**
- **Trigger backup** → runs the relevant cron script immediately
- **Restore from backup** → pick a date/snapshot and restore
- **Verify integrity** → run checksum verification
- **Download archive** link to Wasabi console URL
- **Set retention policy** input
### 2.6 Services (Improve Existing)
**Current:** Systemd service grid, API health cards, SyncroMSP summary.
**Improvements:**
- Add **start / stop / restart** buttons per service
- Show **service logs** (journalctl tail)
- Show **resource usage** per service (memory, CPU, uptime)
- Add **dependencies** — what depends on this service being up
- Add **enable/disable at boot** toggle
- Add **service health history** — uptime %, last restart reason
**Actions:**
- Start service
- Stop service
- Restart service
- View live logs (journalctl -u)
- Enable/disable at boot
- Reload configuration (for services that support it)
### 2.7 Billing & Finance (NEW)
**Purpose:** Business owner's view of revenue, invoices, and accounting sync.
**Data sources:** Stripe, Xero
**Display:**
- **Revenue overview** — MTD, YTD, monthly breakdown (chart)
- **Pending invoices** — outstanding, overdue
- **Payment history** — recent transactions
- **Customer billing summary** — per customer: plan, last payment, balance
- **Subscription health** — active, past-due, canceled counts
- **Accounting sync status** — last Xero sync, any errors
**Actions:**
- View invoice detail
- Send invoice reminder
- Mark invoice as paid (manual override)
- Sync with Stripe
- Sync with Xero
- Export billing report (CSV)
### 2.8 DNS & Domains (NEW dedicated page — or merge into Network)
If kept as separate page:
- All Cloudflare zones with DNS record counts
- Domain registrar details (expiry, auto-renew)
- Quick-add DNS record form
- SSL/TLS settings per domain
- Page Rules / redirect rules
### 2.9 Users & RBAC (NEW)
**Purpose:** Who can access the portal and what they can see/do.
**Display:**
- User list table: name, email, role, last login, status (active/disabled)
- Role/permission matrix view
- Audit log of user actions
**Actions:**
- Create user
- Edit user role
- Disable/delete user
- View user activity log
- Reset password / send invite
### 2.10 Audit Log (NEW)
**Purpose:** Record every action taken in the portal.
**Display:**
- Searchable, filterable timeline of events
- Columns: timestamp, user, action, resource, details, IP
- Exportable to CSV
**Data captures:**
- Who restarted a service
- Who triggered a backup
- Who created a customer
- Who changed a DNS record
- Login/logout events
- Failed permission attempts
### 2.11 Settings (NEW)
**Purpose:** Configure the portal itself and connected integrations.
**Sections:**
- **Integrations** — enable/disable and configure API keys for:
- SyncroMSP
- Cloudflare
- Hetzner
- Stripe
- Xero
- Wasabi
- UniFi
- UISP
- MikroTik
- Hudu
- RingLogix
- RunCloud
- **API Keys** — generate/manage API keys for programmatic portal access
- **Branding** — logo, company name, colors, support phone/email
- **Notifications** — configure alert email/Telegram destinations
- **Backup retention** — set retention policies per bucket
- **Data retention** — how long to keep logs, audit trail, events
### 2.12 Support Tickets (NEW)
**Purpose:** ITPP internal support ticket management.
**Data source:** SyncroMSP
**Display:**
- Ticket queue: ID, customer, subject, status, priority, assignee, last update
- Ticket detail: full conversation, attachments, time tracking
- Ticket metrics: open/closed counts, average response time
**Actions:**
- Create ticket
- Assign ticket
- Update status
- Add note/internal note
- Close/resolve ticket
- Link to customer profile
### 2.13 DRE Claims Dashboard (NEW or link to internal DRE portal)
The DRE portal already exists at `internal.debtrecoveryexperts.com`. On the Ops portal, add a **widget/card** on the dashboard showing:
- Pending approvals count
- Recent claims
- Inbox message count (from dre-mail-poller)
- Link through to the full DRE portal
---
## 3. Actions Per Page
| Page | Read-Only Information | Actions Available |
|------|----------------------|-------------------|
| **Dashboard** | Health summary, recent failures, quick stats | Quick action toolbar, drill into alerts |
| **Servers** | Server specs, status, IP, provider, type | **Restart**, power on/off, create snapshot, WebSSH, view resource graphs |
| **Customers** | Contact info, services, tickets, bills | **Create**, suspend, reactivate, create ticket, view billing |
| **Network** | Router status, interfaces, clients, DNS zones, device health | **Reboot router**, run backup, add DNS record, purge cache, enable/disable interface |
| **Backups** | Bucket status, last upload, age, schedule | **Trigger backup**, **restore from snapshot**, verify integrity, set retention |
| **Services** | Systemd status, API health, SyncroMSP stats | **Start / stop / restart**, view logs, enable/disable boot |
| **Billing** | Revenue, invoices, payments, subscriptions | Sync Stripe, sync Xero, send invoice reminder, export CSV |
| **DNS** | Zone list, record count, SSL status | **Add/Edit/Delete record**, toggle proxy, purge cache |
| **Users/RBAC** | User list, roles, last login | **Create user**, edit role, disable, view audit log |
| **Audit Log** | Full action timeline with user, action, resource | Search, filter, export |
| **Settings** | Integration status, versions, env | **Add/rotate API keys**, enable/disable integrations, set branding |
| **Support Tickets** | Queue, details, metrics | **Create**, assign, update status, add note, resolve |
| **DRE** | Pending approvals, claims, inbox count | Link to full DRE portal |
---
## 4. Integrations Roadmap
### Already Connected & Working
| Integration | Status | Data Flow | Page(s) Using It |
|-------------|--------|-----------|------------------|
| **SyncroMSP** | ✅ Live | 26 customers, tickets, devices via API | Dashboard, Services |
| **Cloudflare** | ✅ Live | 13 zones, DNS record status via API | Network, Dashboard |
| **Hetzner** | ✅ Live | 10 servers, status, type, IP via API | Servers, Dashboard |
| **netcup** | ✅ Live | Server detail (template, IP, id) via API/config | Servers |
| **Wasabi** | ✅ Live | 6 backup buckets, last upload, age via AWS CLI | Backups, Dashboard |
| **Admin-AI** | ✅ Live | API health check | Dashboard, Services |
### In Progress — API Keys Saved, Need Portal UI
| Integration | Status | What to Build |
|-------------|--------|---------------|
| **UniFi** | 🔶 Saved | Add UniFi client count, access point health, client list to Network page |
| **UISP (UNMS)** | 🔶 Saved | Add tower/CPE status, bandwidth to Network page |
### Planned — Need Connection & UI
| Integration | Priority | What to Build |
|-------------|----------|---------------|
| **Stripe** | 🔴 High | Billing page: invoice list, revenue, subscription management |
| **Xero** | 🔴 High | Billing page: accounting sync, P&L snapshot |
| **MikroTik** | 🟡 Medium | Network page: RouterOS API for interface stats, DHCP, firewall |
| **RingLogix** | 🟡 Medium | VoIP status, call metrics, DID inventory, phone management |
| **Hudu** | 🟡 Medium | Link servers/customers to documentation, inline wiki viewer |
| **RunCloud** | 🟢 Low | WordPress server management, site health, SSL |
### Future / Nice-to-Have
| Integration | Use Case |
|-------------|----------|
| **Porkbun / Namecheap** | Domain registration details, expiry dates, auto-renew |
| **GitHub** | Deploy status, latest commits, CI pipeline status |
| **Grafana** | Embed resource graphs, historical metrics |
| **Uptime Kuma / Better Uptime** | External monitoring status, incident history |
| **OpenAI / AI credits** | Usage tracking (Firecrawl already done, can add AI model usage) |
---
## 5. Data Already Flowing
The ops-data-collector (Python, runs every 5 min) already collects:
```json
{
"timestamp": "ISO-8601",
"collection_duration_ms": 10363,
"overall": {
"total_jobs": 21,
"passing": 19,
"failing": 2,
"disk_used_pct": 6,
"memory_used_pct": 41
},
"cron_jobs": [ /* 21 jobs with name, schedule, lastRun, status, script */ ],
"services": { /* 8 systemd services: active/inactive */ },
"disk_memory": { "disk_used_pct", "memory_used_pct" },
"s3_backups": { /* 6 buckets: status, last_upload, age_hours */ },
"api_checks": { /* admin-ai, caddy-config, port checks, cloudflare */ },
"cloudflare_zones": [ /* 13 zone names + status */ ],
"versions": { /* hermes, caddy, python, os */ },
"routers": [ /* currently empty — placeholder */ ],
"netcup_server": { /* id, template, ip */ },
"hetzner_servers": [ /* 10 servers with name, status, type, ip, id */ ],
"syncromsp": { /* customers: 26, tickets, devices, customer_list */ }
}
```
**What the collector CANNOT currently collect:**
- MikroTik RouterOS data (API key needed + collect every 1-5 min)
- Ubiquiti/UISP data (API key saved, but collector not extended yet)
- Stripe billing data (no API key in collector)
- Xero accounting data (no API key in collector)
- Per-server resource usage beyond disk/memory (needs SSH or agent)
- Historical metrics (needs time-series DB like Prometheus or simple log file)
---
## 6. RBAC Recommendations
### 6.1 Roles
| Role | Description |
|------|-------------|
| **Owner** | Full access — see everything, do everything. Germaine. |
| **Admin** | Full access minus billing configuration and system settings. Tony. |
| **Tech / Support** | Servers, services, network, logs, tickets, customers (read). No billing, provisioning, settings. |
| **Sales / Onboarding** | Customer activation flow, order forms, provisioning status. No server internals, no billing. |
| **Viewer / Auditor** | Read-only across limited scope. No actions, no billing, no settings. |
| **DRE Agent** | DRE-specific pages only — claims, inbox, pending approvals. No infrastructure. Anita. |
### 6.2 Page Access Matrix
| Page / Feature | Owner | Admin | Tech/Support | Sales/Onboarding | Viewer | DRE Agent |
|----------------|-------|-------|-------------|-----------------|--------|-----------|
| Dashboard (full) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Dashboard (tech view) | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Dashboard (sales view) | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| Dashboard (DRE view) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Servers — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Servers — restart/power | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Servers — snapshot/provision | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Customers — view | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| Customers — create/edit/suspend | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| Customers — billing detail | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Network — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Network — reboot router/interface | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Network — DNS edit | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Backups — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Backups — trigger/restore | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Services — view | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Services — start/stop/restart | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Billing — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Billing — modify/sync | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Users/RBAC — manage | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Users/RBAC — view | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Audit Log — view | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Settings — all | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Settings — notification prefs | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Tickets — view all | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Tickets — create/assign | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| DRE Claims | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Provisioning (new servers/customers) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
### 6.3 Person-to-Role Mapping
| Person | Role | Notes |
|--------|------|-------|
| **Germaine** (email@redacted) | **Owner** | Sees everything, can do everything. Needs billing, provisioning, and audit. |
| **Tony** (email@redacted) | **Admin** | Full access minus billing config and user management. Handles server operations, backups, customer management. |
| **Anita** | **DRE Agent** | DRE-focused view. Claims, pending approvals, inbox. No infrastructure. |
| **Future Tech Hire** | **Tech/Support** | Servers, services, network, tickets. No billing, provisioning, settings. |
| **Future Sales Hire** | **Sales/Onboarding** | Customer activation, provisioning status, order forms. No server internals. |
| **Future Auditor** | **Viewer** | Read-only, limited scope. Good for compliance or investor review. |
### 6.4 RBAC Implementation Notes
- **Start simple** — use JWT or session tokens with role stored in the token
- **No auth initially** — phase 1: keep open but label sections as "Owner Only" / "Tech Only"
- **Phase 2**: add login with email/password or OAuth (Google/GitHub)
- **Phase 3**: full RBAC enforcement at API/backend level
- Consider using **Hudu's SSO** or **SyncroMSP's user model** as identity source to avoid yet another password
- Store role assignments in a simple JSON config file or small SQLite DB
---
## 7. Implementation Priorities
### Phase 1: Add Actions to Existing Pages (Weeks 1-3)
1. **Backups** — add "Trigger Backup" and "Restore" buttons (calls existing scripts via SSH)
2. **Services** — add Start/Stop/Restart buttons (calls systemctl over SSH)
3. **Servers** — add Reboot + Snapshot buttons (calls Hetzner API)
4. **Network** — add DNS record management (calls Cloudflare API)
### Phase 2: Unlock Placeholders (Weeks 3-5)
5. **MikroTik** — RouterOS API integration, live interface/uptime/leases data
6. **Ubiquiti/UISP** — Connect API, show tower status and client counts
7. **Dashboard improvements** — alert summary, recent activity, quick actions
### Phase 3: New Pages — Foundation (Weeks 5-8)
8. **Customer Management** — list/search/create customers, linked to SyncroMSP
9. **Audit Log** — capture and display every action
10. **Support Tickets** — SyncroMSP ticket queue integration
11. **DRE Dashboard widget** — pending approvals, inbox count
### Phase 4: New Pages — Business & Admin (Weeks 8-12)
12. **Users & RBAC** — login, role assignment, permission enforcement
13. **Billing & Finance** — Stripe revenue, invoice list, Xero sync
14. **Settings** — integration management, API keys, branding
15. **Full RBAC enforcement** — every API endpoint checks permissions
### Phase 5: Polish & Optimization (Ongoing)
16. Add resource graphs (historical CPU/RAM/disk)
17. Add notification preferences per user (email/Telegram for specific alerts)
18. Add dark mode toggle (already dark theme, but add theme options)
19. Mobile-responsive improvements
20. Performance optimization (lazy load, paginate large tables, caching)
---
## Architecture Notes for the Build Team
**Current architecture:**
- Static HTML pages served by Caddy (`ops.itpropartner.com`)
- All data from `ops-status.json` (generated every 5 min by Python collector)
- Vanilla JS fetches JSON and renders DOM
- Shared nav partial (`nav.html`) injected via `fetch()`
- Shared CSS (`ops.css`) and JS (`utils.js`)
- **No backend API** — purely static frontend reading JSON
**Recommended evolution:**
1. Keep static frontend for Phase 1 (quick action buttons can call API endpoints)
2. Add a lightweight backend (FastAPI or Node.js) for:
- Authentication/sessions
- Action endpoints (POST /api/restart-service, POST /api/trigger-backup, etc.)
- Audit logging middleware
- Role/permission checks
3. Options for backend:
- **FastAPI** (Python) — matches existing ecosystem, easy to add SSH/cron exec
- **Express/PocketBase** — simple, built-in auth
- **Remote game** — use Caddy's `reverse_proxy` to a backend running on port 8084
**Data pipeline options for actions:**
- **Direct API call** from backend (best for Hetzner, Cloudflare, Stripe — they have HTTP APIs)
- **SSH execute** (for systemctl on Core and remote servers)
- **Hermes cron trigger** (for backup jobs — trigger the cron script directly)
- **Webhook** (for 3rd party services that support incoming webhooks)
references/restore-plan-2026-07-11.md
+391
View File
@@ -0,0 +1,391 @@
# Restore Plan: Core Server Recovery to 6pm EST (Jul 10, 2026) Working Configuration
> **Generated:** 2026-07-11 07:35 UTC (3:35 AM EST)
> **Prepared by:** Hermes Agent (Sho'Nuff)
> **Status:** READY — validated against live S3 backups and standby server
---
## Executive Summary
The Core server (152.53.192.33) experienced a configuration/model issue around 6pm EST on July 10, 2026. The system is currently running but the daily full backup and root-essentials backup both FAILED overnight (July 11). The July 10 backups are intact and verified. The warm standby (app1-bu, 5.161.114.8) has critical disk space issues (97% used).
**Recovery strategy:** Two paths — (A) In-place fix on Core using verified backups, or (B) Failover to standby after clearing disk space.
---
## Pre-Restore Assessment
### Backup Status
| Backup | Date | Size | Integrity | S3 Path |
|--------|------|------|-----------|---------|
| Full backup | Jul 10, 1:13 AM UTC | 503 MB (2,005 files) | ✅ VALID | `s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz` |
| Root-essentials | Jul 10, 10:38 AM UTC | 42 MB | ✅ VALID | `s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz` |
| Live sync (state.db) | Jul 11, 07:21 UTC | ~1.9 GB | ✅ Current | `s3://hermes-vps-backups/live/state.db` |
| **Full backup Jul 11** | **FAILED** | N/A | ❌ `aws: command not found` | Not uploaded |
| **Root-essentials Jul 11** | **FAILED** | N/A | ❌ Silent failure | Not uploaded |
### Root Cause of Backup Failures
`hermes-backup.sh` (line 215) calls `aws s3 cp` without activating the AWS CLI venv:
```bash
# MISSING from hermes-backup.sh:
source /opt/awscli-venv/bin/activate
```
The `root-essentials-backup.sh` has this line but the backup still failed silently (likely the same `aws` PATH issue or system-level tar path problems). **Fix required before the restore plan can rely on future backups.**
### Warm Standby (app1-bu) Status
| Check | Expected | Actual | Status |
|-------|----------|--------|--------|
| Online | Yes | Up 5d 13h | ✅ |
| Hermes dormant | Inactive | Inactive | ✅ |
| Watchdog cron | `*/5 * * * *` | `*/5 * * * *` | ✅ |
| Sync cron | `*/10 * * * *` | `*/10 * * * *` | ✅ |
| Last sync | Recent | Jul 11 11:30 UTC | ✅ |
| State.db age | Fresh | Jul 10 19:36 UTC | ⚠️ Stale (pre-6pm) |
| Config age | Fresh | Jul 10 21:28 UTC | ⚠️ Stale |
| **Disk space** | >20% free | **3% free (1.2G/38G)** | 🔴 CRITICAL |
### Dependencies Verified
| File | Path | Size | Perms | In Backup? |
|------|------|------|-------|------------|
| config.yaml | /root/.hermes/config.yaml | 3,350 B | 600 | ✅ |
| .env | /root/.hermes/.env | 849 B | 600 | ✅ |
| AWS creds | /root/.aws/credentials | 116 B | 600 | ✅ |
| Caddyfile | /etc/caddy/Caddyfile | 3,094 B | 640 | ✅ |
| SSH key | /root/.ssh/itpp-infra | 399 B | 600 | ✅ |
| Hermes service | /etc/systemd/system/hermes.service | — | — | ✅ |
| Shark-game service | /etc/systemd/system/shark-game.service | — | — | ✅ |
| Ollama service | /etc/systemd/system/ollama.service | — | — | ✅ |
| MySQL tunnel | /etc/systemd/system/mysql-tunnel.service | — | — | ✅ |
### System Health (Current)
| Metric | Value | Status |
|--------|-------|--------|
| Disk | 8% used (448G free) | ✅ |
| Memory | 11 GB available | ✅ |
| Load | 0.16 | ✅ |
| Hermes gateway | Running (PID 1084) | ✅ |
| Anita gateway | Running (PID 935) | ✅ |
| All services | Active/running | ✅ |
---
## Recovery Path A: In-Place Restore from Backup (RECOMMENDED)
Use when Core is running but misconfigured. Restore specific config files from July 10 backups while keeping current state.db and live data.
### Step A1: Restore config.yaml to pre-6pm state
```bash
# Download the root-essentials backup (already verified)
cd /tmp
aws s3 cp s3://hermes-vps-backups/root-backup/root-essentials-2026-07-10.tar.gz \
root-essentials-restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
# Extract just the config file
mkdir -p /tmp/restore-root
tar xzf root-essentials-restore.tar.gz -C /tmp/restore-root
# Back up current config before overwriting
cp /root/.hermes/config.yaml /root/.hermes/config.yaml.pre-restore-$(date +%F-%H%M)
# Restore config.yaml (the pre-6pm version)
cp /tmp/restore-root/.hermes/config.yaml /root/.hermes/config.yaml
chmod 600 /root/.hermes/config.yaml
# Also restore .env if needed
cp /tmp/restore-root/.hermes/.env /root/.hermes/.env
chmod 600 /root/.hermes/.env
# Restore SSH keys if needed
cp /tmp/restore-root/.ssh/* /root/.ssh/
chmod 600 /root/.ssh/*
# Restore AWS credentials if needed
cp /tmp/restore-root/.aws/credentials /root/.aws/credentials
chmod 600 /root/.aws/credentials
```
### Step A2: Restart Hermes gateway to pick up config changes
```bash
hermes gateway restart
```
Wait 5-10 seconds, then verify:
```bash
hermes gateway status
# Expected: Gateway is running (PID: <new>)
```
### Step A3: Verify services are operational
```bash
# Check Hermes responds
hermes terminal --command "echo online"
# Check Telegram connectivity
journalctl -u hermes -n 10 --no-pager | grep -i telegram
# Check all systemd services
systemctl is-active hermes hermes-assistant hermes-browser caddy ollama shark-game mysql-tunnel ops-portal
```
### Step A4: Verify model config works
```bash
# Test model access via admin-ai
KEY=$(grep -A2 'admin-ai:' /root/.hermes/config.yaml | grep api_key | awk '{print $2}')
curl -s https://admin-ai.itpropartner.com/v1/models \
-H "Authorization: Bearer $KEY" | python3 -c "import sys,json;print(len(json.load(sys.stdin).get('data',[])),'models available')"
```
---
## Recovery Path B: Failover to Warm Standby (app1-bu)
Use if Core is unreachable or severely broken. Requires fixing standby disk space first.
### Step B0: PREREQUISITE — Clear disk space on app1-bu (CRITICAL)
The standby has only 1.2 GB free on a 38 GB disk. Failover will fail without space.
```bash
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
# Clean apt cache
apt-get clean
apt-get autoremove --purge -y
# Clean journal logs
journalctl --vacuum-size=200M
# Remove old sync logs if present
> /var/log/hermes-standby-sync.log
# Check what's using space
du -sh /* 2>/dev/null | sort -rh | head -10
# Docker cleanup if docker is installed
docker system prune -a -f 2>/dev/null || true
docker image prune -a -f 2>/dev/null || true
# Verify space recovered
df -h /
# Target: at least 5 GB free
```
### Step B1: Sync latest state before failover
```bash
# From Core: force a live sync to S3
source /opt/awscli-venv/bin/activate
/root/.hermes/scripts/hermes-live-sync.sh
# From app1-bu: pull latest state
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 \
"/root/.hermes/scripts/hermes-standby-sync.sh"
```
### Step B2: Trigger failover
```bash
# On app1-bu: manually activate Hermes
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "
systemctl start hermes 2>/dev/null || \
hermes gateway start
"
```
### Step B3: Verify failover
```bash
# Check gateway status on standby
ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "hermes gateway status"
# Communicate with standby via Telegram
# The watchdog should send Telegram + email alerts when it detects the failover
```
### Step B4: Fencing — prevent split-brain
```bash
# On Core: shut down Hermes gateway to prevent dual-instance conflict
hermes gateway stop
# Or power down Core entirely if it's severely broken
# (Only with user confirmation per reboot-with-health-check protocol)
```
---
## Recovery Path C: Full Rebuild from Full Backup
Use if Core OS is corrupted or server needs to be reprovisioned. Rebuilds everything from the July 10 full backup tarball.
### Step C1: Provision replacement server (if needed)
If on netcup: use netcup SCP to provision a new RS 2000.
If moving to Hetzner: provision CPX41 or similar (8C/16G minimum).
### Step C2: Install base packages
```bash
apt-get update && apt-get install -y \
python3 python3-pip python3-venv git curl wget \
caddy docker.io docker-compose awscli sqlite3 \
wireguard strongswan xl2tpd ollama
```
### Step C3: Install Hermes
```bash
pip3 install hermes-agent
```
### Step C4: Download and extract full backup
```bash
cd /tmp
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-10.tar.gz \
full-restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
tar xzf full-restore.tar.gz -C /tmp/restore
```
### Step C5: Restore all components
```bash
# Hermes config + state
cp -r /tmp/restore/hermes-backup-2026-07-10/* /root/.hermes/ 2>/dev/null || true
cp /tmp/restore/hermes-backup-2026-07-10/config/config.yaml /root/.hermes/
cp /tmp/restore/hermes-backup-2026-07-10/config/.env /root/.hermes/
# SSH keys
cp /tmp/restore/hermes-backup-2026-07-10/ssh/* /root/.ssh/ 2>/dev/null || true
# AWS credentials
cp /tmp/restore/hermes-backup-2026-07-10/aws/* /root/.aws/ 2>/dev/null || true
# Mail passwords
cp -r /tmp/restore/hermes-backup-2026-07-10/himalaya/* /root/.config/himalaya/ 2>/dev/null || true
# Caddyfile
cp /tmp/restore/hermes-backup-2026-07-10/caddy/Caddyfile /etc/caddy/Caddyfile
# Systemd services
cp /tmp/restore/hermes-backup-2026-07-10/systemd/*.service /etc/systemd/system/
# Set permissions
chmod 600 /root/.hermes/config.yaml /root/.hermes/.env /root/.aws/credentials
chmod 600 /root/.ssh/*
chmod 640 /etc/caddy/Caddyfile
```
### Step C6: Start services
```bash
systemctl daemon-reload
systemctl enable --now hermes caddy ollama shark-game mysql-tunnel ops-portal
```
### Step C7: Run the embedded restore script
```bash
bash /tmp/restore/hermes-backup-2026-07-10/restore.sh
```
---
## Post-Restore Verification Checklist
After ANY restore path, complete these checks:
### 1. Hermes Core
- [ ] `hermes gateway status` — Gateway is running
- [ ] `hermes terminal --command "echo online"` — Agent responds
- [ ] Telegram bot responds to `/status`
### 2. Web Services
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://core.itpropartner.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://portal.debtrecoveryexperts.com` → 200
- [ ] `curl -s -o /dev/null -w "%{http_code}" https://shark.iamgmb.com` → 200
### 3. Model Access
- [ ] `curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $KEY" | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if 'data' in d else 'FAIL')"` → OK
### 4. Email
- [ ] `himalaya envelope list -s 5` — IMAP access works
### 5. Backups
- [ ] `aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com | tail -3` — Recent backups exist
- [ ] `aws s3 ls s3://hermes-vps-backups/live/state.db --endpoint-url https://s3.us-east-1.wasabisys.com` — Live sync is current
### 6. Standby
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "uptime"` — Standby reachable
- [ ] `ssh -i /root/.ssh/itpp-infra root@5.161.114.8 "df -h / | tail -1"` — Disk >10% free
---
## Critical Fix: Repair Backup Script for Future Reliability
The daily backup script MUST be fixed to prevent recurrence:
```bash
# Add AWS CLI venv activation to hermes-backup.sh
```
Add after line 5 (`set -euo pipefail`):
```bash
# Activate AWS CLI from venv (added 2026-07-11 — DR FIX)
if [ -f /opt/awscli-venv/bin/activate ]; then
source /opt/awscli-venv/bin/activate
fi
```
Similarly verify `root-essentials-backup.sh` works end-to-end:
```bash
# Test run root-essentials backup manually
bash -x /root/.hermes/scripts/root-essentials-backup.sh 2>&1
```
Then update the DR issue log with this finding.
---
## Team Contact & Escalation
| Role | Name | Responsibility |
|------|------|---------------|
| CEO | Germaine | Final decision authority |
| Lead Sysadmin | Sho'Nuff (Hermes) | First responder, triage, fix |
| Operations | Anita | Comms, client notification |
**Escalation:** If restore fails after 2 attempts → escalate to netcup support (Core) or Hetzner support (standby).
---
## Rollback Plan
If restore makes things worse:
1. Restore config from pre-restore backup: `cp /root/.hermes/config.yaml.pre-restore-* /root/.hermes/config.yaml`
2. `hermes gateway restart`
3. Verify system returns to current state (which was operational but misconfigured)
---
## Files Created/Modified During Assessment
| File | Action |
|------|--------|
| This restore plan | Created at `/root/.hermes/references/restore-plan-2026-07-11.md` |
| `/tmp/root-verify.tar.gz` | Downloaded root-essentials backup for integrity check (clean up after) |
| `/tmp/full-verify.tar.gz` | Downloaded full backup for integrity check (clean up after) |
| DR issue log | New entry needed for backup script AWS CLI PATH bug |
references/restore-runbooks.md
+324
View File
@@ -0,0 +1,324 @@
# Standalone Restore Runbooks
## 1. Primary Host (Core) — Hermes
**Estimated time to complete:** 15 minutes
**Rollback plan:** Revert DNS to standby `app1-bu` (5.161.114.8). Wipe server and restart runbook.
### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials (for Wasabi)
- netcup panel access
### Step 1: Provision server
Order a **netcup RS 2000 G12** (8C EPYC, 16GB RAM, 512GB NVMe) running Debian 13.
### Step 2: Base install
```bash
hostnamectl set-hostname core
apt update && apt upgrade -y
apt install -y fail2ban prometheus-node-exporter ufw docker.io docker-compose-v2
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
useradd -m -G sudo ippadmin
mkdir -p /home/ippadmin/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
```
### Step 3: Restore data
```bash
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env --endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/core/latest/docker/ /root/docker/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```
### Step 4: Start services
```bash
for dir in /root/docker/*/; do
(cd "$dir" && docker compose up -d)
done
systemctl restart caddy
hermes gateway run --replace
```
### Step 5: Verify
```bash
curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Open Telegram and send /ping to Hermes bot; ensure it replies immediately.
```
---
## 2. AI Server (admin-ai) — LiteLLM + OpenWebUI + Ollama
**Estimated time to complete:** 2 hours
**Rollback plan:** Use external model providers in LiteLLM config or fallback to Core's local Ollama. Wipe server and retry.
### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access
### Step 1: Provision server
Order a **Hetzner CPX41** running Debian 12 (or 13).
### Step 2: Base install
```bash
hostnamectl set-hostname admin-ai
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y ufw docker-compose-plugin
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
```
### Step 3: Restore data
```bash
aws s3 cp s3://hermes-vps-backups/admin-ai/latest/opt/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```
### Step 4: Start services
```bash
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
# Update DNS to point admin-ai.itpropartner.com to the new IP address
```
### Step 5: Verify
```bash
# Wait for DNS propagation, then test completion API
curl https://admin-ai.itpropartner.com/v1/chat/completions \
-H "Authorization: Bearer $(grep ADMIN_AI ~/.hermes/.env | cut -d= -f2)" \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Log into OpenWebUI via browser to ensure UI and database are intact
```
---
## 3. Legacy Web Host (wphost02) — WordPress + RunCloud
**Estimated time to complete:** 4 hours
**Rollback plan:** Delete the broken Hetzner VPS instance, revert DNS. Restore an older snapshot in RunCloud.
### Prerequisites
- RunCloud panel access
- Hetzner Cloud panel access
- Credentials for `email@redacted` Postfix relay
### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
### Step 2: Base install
```bash
# Connect to RunCloud panel and add the new server IP.
# RunCloud will automatically provision nginx, MariaDB, and PHP.
# Deploy itpp-infra SSH key in the RunCloud panel for SSH access.
# In RunCloud firewall, whitelist Core IP: 152.53.192.33
```
### Step 3: Restore data
```bash
# Go to RunCloud panel > Web Applications.
# Restore the WordPress site from the RunCloud daily backup.
# Manually restore the database if the backup does not include it.
```
### Step 4: Start services
```bash
# Web services start automatically via RunCloud. Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:PASSWORD_HERE" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
postqueue -f
```
### Step 5: Verify
```bash
curl -I https://apex-vreg-live.com # Or target domain, check for 200 OK
# Verify email delivery by triggering a password reset or test email from WordPress.
```
---
## 4. UNMS (UISP) — Docker stack
**Estimated time to complete:** 4 hours
**Rollback plan:** `docker compose down -v`, clear the volume path, and re-pull the backup from an older date.
### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access
### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
### Step 2: Base install
```bash
hostnamectl set-hostname unms
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 2055/udp && ufw --force enable
```
### Step 3: Restore data
```bash
mkdir -p /opt/unms
aws s3 cp s3://hermes-vps-backups/unms/latest/ /opt/unms/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```
### Step 4: Start services
```bash
cd /opt/unms
docker compose up -d
# Update DNS to point the UISP domain to the new IP address.
```
### Step 5: Verify
```bash
# Navigate to the UNMS/UISP web interface.
# Log in and confirm the device list populates and shows towers/routers as connected.
# Check container health:
docker ps | grep unms
```
---
## 5. UniFi — Controller
**Estimated time to complete:** 4 hours
**Rollback plan:** Factory reset UniFi via `dpkg-reconfigure unifi` and upload an older `.unf` backup file.
### Prerequisites
- A recent UniFi `.unf` backup file (from S3 or local download)
- Hetzner Cloud panel access
### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
### Step 2: Base install
```bash
hostnamectl set-hostname unifi
apt update && apt upgrade -y
apt install -y ca-certificates apt-transport-https openjdk-17-jre-headless ufw
ufw allow ssh && ufw allow 8080 && ufw allow 8443 && ufw allow 3478/udp && ufw --force enable
```
### Step 3: Restore data
```bash
# Add Ubiquiti repo and install UniFi (native, no Docker)
echo 'deb https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/100-ubnt-unifi.list
wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg
apt update && apt install -y unifi
```
### Step 4: Start services
```bash
systemctl enable --now unifi
# Wait for the service to start (can take a few minutes)
# Access the web UI at https://<new-ip>:8443
# Select "Restore from a previous backup" and upload the .unf file
```
### Step 5: Verify
```bash
systemctl status unifi
# Log into the UniFi UI
# Confirm access points and switches show as "Online" (Adopted).
```
---
## 6. n8n — Docker
**Estimated time to complete:** 8 hours
**Rollback plan:** Stop Docker, delete `/opt/n8n`, sync an older backup snapshot from S3.
### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access
### Step 1: Provision server
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.
### Step 2: Base install
```bash
hostnamectl set-hostname n8n
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw --force enable
```
### Step 3: Restore data
```bash
mkdir -p /opt/n8n
aws s3 cp s3://hermes-vps-backups/n8n/latest/ /opt/n8n/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```
### Step 4: Start services
```bash
cd /opt/n8n
docker compose up -d
# Update DNS to point the n8n domain to the new IP address.
```
### Step 5: Verify
```bash
# Access the n8n web interface.
# Check that all workflows are present.
# Execute a test run of a non-critical workflow to verify Postgres connectivity.
```
---
## 7. Docker Box — Compose stacks
**Estimated time to complete:** 8 hours
**Rollback plan:** Run `docker compose down`, remove volume data, download an older backup archive, and try starting again.
### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access
### Step 1: Provision server
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.
### Step 2: Base install
```bash
hostnamectl set-hostname docker-box
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 21115:21119/tcp && ufw allow 21116/udp && ufw --force enable
```
### Step 3: Restore data
```bash
mkdir -p /opt
aws s3 cp s3://hermes-vps-backups/docker-box/latest/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```
### Step 4: Start services
```bash
# Start all 8 containers (NPM, RustDesk, Uptime Kuma, RunCloud, etc.)
for dir in /opt/*/; do
if [ -f "$dir/docker-compose.yml" ]; then
(cd "$dir" && docker compose up -d)
fi
done
# Update DNS records for NPM, Uptime Kuma, and RustDesk relay to the new IP.
```
### Step 5: Verify
```bash
# Log into Uptime Kuma web UI and verify monitors are running.
# Access Nginx Proxy Manager (NPM) UI and check proxy hosts.
# Open a RustDesk client and ensure it connects to the self-hosted relay.
```
references/server-dr-plans-public.md
+495
View File
@@ -0,0 +1,495 @@
# Company Name — Disaster Recovery Plan v3 (Consolidated)
**Author:** [Lead Sysadmin] / Network Services Team, incorporating third-party DR review
**Date:** July 9, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
---
## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses
---
## 1. Architecture & Shared Dependencies
```
Internet ──► CDN/DNS Provider ──► Caddy ──► Service (HTTP/HTTPS)
│
┌───────────┼───────────┐
│ │ │
SSH/WG Tailscale Management
(company-prefix- (Primary Host ↔ VPN (WireGuard
infra) Standby Host) to home router)
│ │ │
└───────────┼───────────┘
│
┌───────────┴───────────┐
│ │
S3 Primary S3 Secondary
(Object Storage Provider us-east-1) (Object Storage Provider us-west-2
versioning ON or Backblaze B2)
object lock ON object lock ON
```
### Shared Dependencies — Documented
| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | company-prefix-infra key (Secondary Provider vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | CDN/DNS Provider (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Object Storage Provider us-east-1 | Object Storage Provider us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Object Storage Provider us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | [MAIL-SERVER-DOMAIN]:2525 | Direct Email Provider (port 465/587) | ~/.hermes/.env |
| Auth/SSO | CDN/DNS Provider Access | Local portal auth fallback | CDN/DNS Provider dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
---
### 3.1 Backup Matrix
| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE (new) |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE (new) |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | ⚠️ NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | ⚠️ PARTIAL |
| S3 secondary | Daily sync | Object Storage Provider us-west-2 | 90 days | ❌ NOT SET UP |
### 3.3 Backup Security
- **Versioning:** ON (all 3 buckets) ✅
- **Object Lock:** NOT enabled — needs implementation
- **Delete protection:** NOT configured
- **Backup credentials:** Write-only IAM user — NOT implemented (using admin credentials)
- **Encryption at rest:** Object Storage Provider server-side (default) — no client-side encryption
- **Credentials escrow:** Partial (in .env + Hudu)
**Fixes needed:**
1. Create IAM sub-user with `s3:PutObject` only (no delete, no list)
2. Enable Object Lock with 7-day minimum retention
3. Store write-only credentials separately from admin credentials
4. Client-side encrypt backup before upload (age/gpg) for the secondary copy
---
## 4. Failover / Failback
### 4.1 Warm Standby — Corrected Timing
The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.
**Old:** Check every 5 min, failover after 2 min unreachable
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min)
```python
# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30 # seconds
FAILURE_THRESHOLD = 4 # consecutive failures
CONFIRMATION_WINDOW = 120 # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)
```
### 4.2 Health Check — Two Layers
**Layer 1 (Ping):** ICMP to Primary Host ([IP-REDACTED]) every 30s
**Layer 2 (HTTP):** `GET /[GENERIC-HEALTH-ENDPOINT]` on Tailscale IP — checks:
- Gateway heartbeat ≤ 5s old
- SQLite `PRAGMA quick_check` returns "ok"
- Local Ollama responding
**Decision Matrix:**
| Server | Hermes | Action |
|---|---|---|
| ✅ UP | ✅ UP | Stay dormant |
| ❌ DOWN | N/A | **Auto-takeover** — unambiguous |
| ✅ UP | ❌ DOWN | **ALERT + fence** — attempt SSH fence, only takeover if Primary Host Hermes confirmed stopped |
### 4.3 Fencing Protocol
```bash
# SSH Fence (preferred — server is up)
ssh [USER-REDACTED]@[IP-REDACTED] \
"sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh [USER-REDACTED]@[IP-REDACTED] "sudo systemctl is-active hermes"
# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Primary Host refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Primary Host reads on startup — if standby owns it, Primary Host stays dormant
```
### 4.4 Failback Protocol
**⚠️ Auto-failback is FORBIDDEN.** Human clearance required.
```bash
# FAILBACK STEPS (manual):
# 1. Confirm Primary Host is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh Standby Host "hermes gateway stop"
# 3. Sync state from standby back to Primary Host (if needed):
# Rsync /root/.hermes/ from standby to Primary Host
# 4. Unmask Hermes on Primary Host:
# ssh Primary Host "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Primary Host gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"
```
### 4.5 Provider/Account Outage Scenario
If the entire hosting provider (Primary Provider) is unavailable:
1. Standby Host (Secondary Provider) takes over as active Hermes host
2. DNS records updated to point to Standby Host IP ([IP-REDACTED])
3. Services migrate to Primary Provider replacement when available
4. No auto-failback — human decision
---
## 5. Monitoring & Alerting
### 5.1 Alert Thresholds
| Alert | Threshold | Action | Priority |
|---|---|---|---|
| Disk usage | > 80% | Notify, investigate | 🟡 P2 |
| Disk usage | > 90% | Notify, immediate cleanup | 🔴 P0 |
| Backup age | > 24h since last success | Notify | 🟡 P1 |
| Backup failure | Any failed job | Notify | 🔴 P0 |
| Hermes heartbeat | No heartbeat > 30s | Notify | 🔴 P0 |
| Hermes down + server up | Layer 2 fail, Layer 1 pass | IMMEDIATE alert | 🔴 P0 |
| TLS expiration | < 14 days | Notify | 🟡 P1 |
| TLS expiration | < 7 days | Notify + remind | 🔴 P0 |
| SSH fail spikes | > 10 failures/min | Investigate | 🟡 P1 |
| Service down | Caddy/nginx/Docker inactive | Notify, attempt restart | 🔴 P0 |
| S3 sync failure | Upload fails | Abort (memory consolidate) | 🔴 P0 |
| Standby stale | No sync > 30 min | Notify | 🟡 P1 |
### 5.2 Monitoring Stack
- **Per-server:** `prometheus-node-exporter` (all boxes)
- **Dashboard:** Prometheus → Grafana ()
- **Synthetic checks:** Uptime Kuma (on docker box, [IP-REDACTED])
- **Hermes watchdog:** `service-health-check.sh` (every 5 min, Primary Host)
- **Standby watcher:** `home-router-watchdog.sh` (every 5 min, Primary Host)
- **Apex mail:** `apex-mail-watchdog.sh` (every 5 min, Primary Host)
- **Alert delivery:** Telegram (via Hermes)
## 6. Restore Runbooks (Per Server)
| Attribute | Detail |
|---|---|
| RTO target | 5 minutes (via failover to Standby Host) |
| RPO target | 15 minutes (Hermes state), 24h (Docker data) |
| Actual RTO | ~3 minutes (corrected watchdog) |
| Actual RPO | Varies by data type (see below) |
**Restore steps:**
```bash
# Option A: Failover to standby (fastest — < 5 min)
# Standby Host detects Primary Host down → auto-takeover (Section 4.1)
# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on Primary Provider
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
--endpoint-url https://[S3-STORAGE-DOMAIN]
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
--endpoint-url https://[S3-STORAGE-DOMAIN]
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
--endpoint-url https://[S3-STORAGE-DOMAIN]
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
(cd "dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://[OPS-PORTAL-DOMAIN]/[GENERIC-HEALTH-ENDPOINT]
curl -s https://[DOCUMENT-SIGNING-DOMAIN]
# Check Telegram responds to /ping
```
### 6.2 [AI-SERVICE-DOMAIN] (Secondary Provider CPX41) — 🔴 Critical
| Attribute | Detail |
|---|---|
| RTO target | 2 hours |
| RPO target | 24 hours |
| Current disk | **🔴 92%** — must clean before migration |
**⚠️ Pre-reboot checklist:**
```bash
# Verify at least one alternative model provider works
curl https://[AI-ADMIN-DOMAIN]/[GENERIC-LLM-ENDPOINT] \
-H "Authorization: Bearer (cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Primary Host is running
curl -s http://[IP-REDACTED]:11434/[GENERIC-OLLAMA-ENDPOINT] || systemctl start ollama
# Notify all Hermes users ([CEO], [Operations Manager], [Developer])
```
**Restore steps:**
```bash
# Step 1 — Provision replacement (CPX41 or Primary Provider equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: [AI-ADMIN-DOMAIN] → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles
```
### 6.3 Legacy Web Host (Secondary Provider CPX21, Server Management Panel) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Backup method | Server Management Panel daily backup + manual DB |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 on Secondary Provider
# Step 2 — Connect to Server Management Panel panel, link server
# Step 3 — Server Management Panel provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from Server Management Panel backup → verify apex site loads
# Step 5 — Deploy company-prefix-infra SSH key in Server Management Panel → SSH Access
# Step 6 — Whitelist Primary Host IP ([IP-REDACTED]) in Server Management Panel firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = [MAIL-SERVER-DOMAIN]:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "[MAIL-SERVER-DOMAIN]:2525 [EMAIL-REDACTED]:[CREDENTIAL-REDACTED]" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f
```
### 6.4 UNMS (Secondary Provider CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 9 (Docker compose stack) |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS
```
### 6.5 Hudu (Secondary Provider CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 5 (Docker compose) |
| Auto-backup | Hudu built-in S3 backup |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible
```
### 6.6 UniFi (Secondary Provider CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Type | Native install (not Docker) |
**Restore steps:**
```bash
# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches
```
### 6.7 n8n (Secondary Provider CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 2 (n8n + postgres) |
**Restore:**
```bash
# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d
```
### 6.8 docker box (Secondary Provider CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 8 (NPM, RustDesk, Uptime Kuma, Server Management Panel) |
**Restore:**
```bash
# Provision → Docker → restore compose stacks + volumes
```
### 6.9 fleettracker360 (Secondary Provider CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Type | Apache + MariaDB (LAMP) |
**Restore:**
```bash
# Provision → Apache + MariaDB → restore web root + DB
```
### 6.10 developer-vps (Secondary Provider CPX21) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | Coordinate with [Developer] |
| Type | Independent Hermes + local LLM |
**⚠️ SHADOW IT — needs documentation.**
**Current known config:**
- Hermes gateway with admin-ai model access
- Llama 3.2 1B local fallback (systemd: llama-server)
- Cron: hermes-heartbeat.sh every minute
- Minimal services, no Docker
**Actions needed:**
1. Document [Developer]'s full Hermes config.yaml and .env
2. Backup to S3 under s3://hermes-vps-backups/developer-vps/
3. Add [Developer]'s config to DR plan once documented
### 6.11 Standby Host (Secondary Provider CPX21) — 🔴 Standby
| Attribute | Detail |
|---|---|
| RTO target | 2 minutes |
| RPO target | 15 minutes |
| Type | Warm standby — gateway dormant |
**Activation:**
```bash
# Automatic: watchdog detects Primary Host down → S3 sync → start gateway
# Manual:
ssh [USER-REDACTED]@[IP-REDACTED]
sudo -i
hermes gateway run --replace
```
---
## 7. Testing & Validation
### 7.1 Restore Testing Schedule
| Test | Frequency | Who |
|---|---|---|
| Verify backup job completion (all servers) | Daily | Automated alert |
| Check backup age per server | Daily | Automated |
| Restore one database dump to test environment | Monthly | Network Team |
| Restore one full application stack | Quarterly | Network Team |
| Full DR simulation — Primary Host failover to Standby Host | Quarterly | Network Team |
| S3 credential restore/access test | Quarterly | Network Team |
| Document restore test results (date, tester, duration, findings) | Per test | Network Team |
### 7.2 Validation Criteria
Each restore test must produce:
- Date + tester name
- Server/application tested
- Backup artifact used (path + timestamp)
- Restore duration (start → fully operational)
- Data validation: check a known record exists (specific row, file, or document)
- Issues encountered
- Remediation ticket (if needed)
### 7.3 Quarterly DR Simulation
Scenario: "Primary Host is completely gone. Primary Provider account inaccessible. Standby Host has been active for 4 hours. Now Primary Provider is back — fail back."
Steps to validate:
1. Simulate Primary Provider provider outage (block Primary Host IP)
2. Verify Standby Host takes over within 5 minutes
3. Send a Telegram message through standby
4. Verify all cron jobs run on standby
5. Simulate Primary Provider recovery
6. Execute manual failback per Section 4.4
7. Verify no data loss, no split-brain
---
## 8. Security
### 8.1 Current Security Posture
| Control | Status | Notes |
|---|---|---|
| SSH keys only | ✅ Done | Password auth disabled |
| Root SSH disabled | ⚠️ Partial | Some servers still allow root |
| UFW host firewall | ✅ All | Standard on all boxes |
| fail2ban | ✅ All | Standard on all boxes |
| S3 versioning | ✅ All buckets | Enabled |
| S3 Object Lock | ❌ Not done | Needs implementation |
| S3 write-only IAM | ❌ Not done | Currently using admin creds |
| Backup encryption | ❌ Not done | Server-side only; no client-side |
| Secrets escrow | 🟡 Partial | .env + Hudu — not all secrets documented |
| TLS everywhere | ✅ All public endpoints | Caddy auto-certs |
### 8.3 Break-Glass Access
If company-prefix-infra key is lost or revoked:
1. **Secondary Provider servers:** Use Secondary Provider console → Rescue mode → mount filesystem → redeploy key
2. **Primary Provider servers:** Use SCP web console → VNC → log in with root password → redeploy key
3. **Root passwords:** Stored in ~/.hermes/.env (chmod 600) and Hudu
---
references/server-dr-plans-redacted.md
+98
View File
@@ -0,0 +1,98 @@
# IT Pro Partner — Per-Server Disaster Recovery Plan (REDACTED)
**Author:** Sho'Nuff / Network Services Team
**Date:** July 9, 2026
**Version:** 1.0 — Redacted for 3rd Party Review
---
## Architecture Overview
```
Internet ──► CDN ──► Reverse Proxy ──► Service (HTTP/HTTPS)
│
SSH/WG ──► Management plane (key-based auth)
│
S3-Compatible Storage ──► Backup plane (all servers)
```
**Shared dependencies:**
- **SSH access:** Ed25519 key, deployed to all servers
- **DNS:** Cloudflare (primary) / legacy provider (one domain)
- **Backup storage:** S3-compatible (Wasabi, us-east-1)
- **SMTP relay:** Authenticated relay (STARTTLS, port 2525)
- **Auth/SSO:** Cloudflare Access (internal portals)
- **Monitoring:** Prometheus node_exporter per server
---
## Server Inventory Summary
**11 servers total:**
- 1 Primary operations server (netcup, dedicated EPYC)
- 1 AI/LLM inference server (Hetzner)
- 1 WordPress/RunCloud server (Hetzner)
- 3 Application servers (Hetzner — UISP/UNMS, Hudu, UniFi)
- 3 Utility servers (Hetzner — n8n, Docker staging, fleet tracking)
- 1 Standalone Hermes server (Hetzner — Tony)
- 1 Warm standby server (Hetzner)
---
## DR Classification
| Server | Tier | RTO Target | RPO Target | Restore Complexity |
|--------|------|-----------|-----------|-------------------|
| Primary Ops | Critical | 5 min (failover) | 15 min | Medium |
| AI/LLM | Critical | 2 hours | 24 hours | High |
| WordPress | Important | 4 hours | 24 hours | Low |
| UISP/UNMS | Important | 4 hours | 24 hours | Medium |
| IT Documentation | Important | 4 hours | 24 hours | Medium |
| Network Controller | Important | 4 hours | 24 hours | Low |
| Automation (n8n) | Standard | 8 hours | 24 hours | Low |
| Docker Utility | Standard | 8 hours | 24 hours | Medium |
| Fleet Tracking | Standard | 8 hours | 24 hours | Low |
| Standalone Agent | Standard | Coordinate | 24 hours | Low |
| Warm Standby | Standby | 2 min | 15 min | Low |
---
## Backup Strategy
| Data | Frequency | Location | Retention |
|------|-----------|----------|-----------|
| System config | Daily | S3-compatible storage | 90 days |
| Docker volumes | Daily | S3-compatible storage | 90 days |
| Database dumps | Daily | S3-compatible storage | 90 days |
| Application data | Per-application | S3-compatible storage | 90 days |
| Agent state | Every 15 min | S3-compatible storage | 48 hours |
---
## Restore Procedure (Generic)
```bash
# 1. Provision replacement server (same tier or better)
# 2. Install Debian (current stable)
# 3. Deploy SSH key
# 4. Install Docker (if applicable)
# 5. Restore from backup storage:
aws s3 cp s3://backup-bucket/<server>/latest/ /opt/restore/ --recursive
# 6. Restore Docker stacks
cd /opt/<service> && docker compose up -d
# 7. Verify health endpoints return 200
```
---
## Key Recommendations from Audit
1. **AI server disk critical (92% full):** Needs cleanup before any migration
2. **Docker volumes not backed up to S3:** All Docker hosts need volume backup automation
3. **Database dumps not automated:** MariaDB on wphost02, fleettracker360 need scheduled dumps
4. **Tony's Hermes:** Independent instance — document its config separately
5. **Migration plan:** Consolidate Hetzner services to netcup per the server standard
---
*Full version with IPs, credentials references, and detailed restore commands is stored internally and available on request.*
references/server-dr-plans-v3.md
+655
View File
@@ -0,0 +1,655 @@
# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)
**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
**Date:** July 9, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
---
## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses
---
## 1. Architecture & Shared Dependencies
```
Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
│
┌───────────┼───────────┐
│ │ │
SSH/WG Tailscale Management
(itpp- (Core ↔ VPN (WireGuard
infra) app1-bu) to home router)
│ │ │
└───────────┼───────────┘
│
┌───────────┴───────────┐
│ │
S3 Primary S3 Secondary
(Wasabi us-east-1) (Wasabi us-west-2
versioning ON or Backblaze B2)
object lock ON object lock ON
```
### Shared Dependencies — Documented
| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
---
## 2. Server Standards
### 2.1 Tier Definitions
| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |
### 2.2 Base Install — Verified Checklist
```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York
# S2: system update
apt update && apt upgrade -y
# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable
# S4: monitoring
apt install -y prometheus-node-exporter
# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd
# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
```
### 2.3 Application Data Paths — Per Server
Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies
(See Section 6 — Restore Runbooks for per-server details)
---
## 3. Backup Strategy
### 3.1 Backup Matrix
| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE (new) |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE (new) |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | ⚠️ NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | ⚠️ PARTIAL |
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |
### 3.2 Immediate Backup Gaps to Close
| Priority | Gap | Fix |
|---|---|---|
| 🔴 P0 | Docker volumes not backed up | Deploy volume backup script per Docker host |
| 🔴 P0 | Database dumps not automated | Cron job per MariaDB/Postgres instance |
| 🟡 P1 | No secondary S3 target | Provision Wasabi us-west-2 bucket + replication |
| 🟡 P1 | App data recovery undocumented | Per-app data path inventory (Section 2.3) |
| 🟢 P2 | 48h snapshot retention too short | Extend to 30 days via S3 lifecycle policy |
| 🟢 P2 | System config backup not automated | Add `/etc/` snapshot to daily backup cron |
### 3.3 Backup Security
- **Versioning:** ON (all 3 buckets) ✅
- **Object Lock:** NOT enabled — needs implementation
- **Delete protection:** NOT configured
- **Backup credentials:** Write-only IAM user — NOT implemented (using admin credentials)
- **Encryption at rest:** Wasabi server-side (default) — no client-side encryption
- **Credentials escrow:** Partial (in .env + Hudu)
**Fixes needed:**
1. Create IAM sub-user with `s3:PutObject` only (no delete, no list)
2. Enable Object Lock with 7-day minimum retention
3. Store write-only credentials separately from admin credentials
4. Client-side encrypt backup before upload (age/gpg) for the secondary copy
---
## 4. Failover / Failback
### 4.1 Warm Standby — Corrected Timing
The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.
**Old:** Check every 5 min, failover after 2 min unreachable
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min)
```python
# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30 # seconds
FAILURE_THRESHOLD = 4 # consecutive failures
CONFIRMATION_WINDOW = 120 # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)
```
### 4.2 Health Check — Two Layers
**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
**Layer 2 (HTTP):** `GET /healthz` on Tailscale IP — checks:
- Gateway heartbeat ≤ 5s old
- SQLite `PRAGMA quick_check` returns "ok"
- Local Ollama responding
**Decision Matrix:**
| Server | Hermes | Action |
|---|---|---|
| ✅ UP | ✅ UP | Stay dormant |
| ❌ DOWN | N/A | **Auto-takeover** — unambiguous |
| ✅ UP | ❌ DOWN | **ALERT + fence** — attempt SSH fence, only takeover if Core Hermes confirmed stopped |
### 4.3 Fencing Protocol
```bash
# SSH Fence (preferred — server is up)
ssh ippadmin@152.53.192.33 \
"sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh ippadmin@152.53.192.33 "sudo systemctl is-active hermes"
# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Core refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Core reads on startup — if standby owns it, Core stays dormant
```
### 4.4 Failback Protocol
**⚠️ Auto-failback is FORBIDDEN.** Human clearance required.
```bash
# FAILBACK STEPS (manual):
# 1. Confirm Core is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh app1-bu "hermes gateway stop"
# 3. Sync state from standby back to Core (if needed):
# Rsync /root/.hermes/ from standby to Core
# 4. Unmask Hermes on Core:
# ssh Core "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Core gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"
```
### 4.5 Provider/Account Outage Scenario
If the entire hosting provider (netcup) is unavailable:
1. app1-bu (Hetzner) takes over as active Hermes host
2. DNS records updated to point to app1-bu IP (5.161.114.8)
3. Services migrate to netcup replacement when available
4. No auto-failback — human decision
---
## 5. Monitoring & Alerting
### 5.1 Alert Thresholds
| Alert | Threshold | Action | Priority |
|---|---|---|---|
| Disk usage | > 80% | Notify, investigate | 🟡 P2 |
| Disk usage | > 90% | Notify, immediate cleanup | 🔴 P0 |
| Backup age | > 24h since last success | Notify | 🟡 P1 |
| Backup failure | Any failed job | Notify | 🔴 P0 |
| Hermes heartbeat | No heartbeat > 30s | Notify | 🔴 P0 |
| Hermes down + server up | Layer 2 fail, Layer 1 pass | IMMEDIATE alert | 🔴 P0 |
| TLS expiration | < 14 days | Notify | 🟡 P1 |
| TLS expiration | < 7 days | Notify + remind | 🔴 P0 |
| SSH fail spikes | > 10 failures/min | Investigate | 🟡 P1 |
| Service down | Caddy/nginx/Docker inactive | Notify, attempt restart | 🔴 P0 |
| S3 sync failure | Upload fails | Abort (memory consolidate) | 🔴 P0 |
| Standby stale | No sync > 30 min | Notify | 🟡 P1 |
### 5.2 Monitoring Stack
- **Per-server:** `prometheus-node-exporter` (all boxes)
- **Dashboard:** Prometheus → Grafana (planned — on app1 or dedicated)
- **Synthetic checks:** Uptime Kuma (on docker box, 178.156.168.35)
- **Hermes watchdog:** `service-health-check.sh` (every 5 min, Core)
- **Standby watcher:** `home-router-watchdog.sh` (every 5 min, Core)
- **Apex mail:** `apex-mail-watchdog.sh` (every 5 min, Core)
- **Alert delivery:** Telegram (via Hermes)
### 5.3 Alerting Gaps to Close
| Gap | Fix |
|---|---|
| No Prometheus/Grafana stack | Deploy on app1 after migration |
| No backup success/failure dashboard | Add to ops portal |
| No disk threshold alerts for all servers | Add to ops-data-collector.py |
| No TLS expiry monitoring | Add cert check to service-health-check.sh |
| No centralized alert routing | Use Hermes for all alerts |
---
## 6. Restore Runbooks (Per Server)
### 6.1 Core (netcup RS 2000) — 🔴 Critical
| Attribute | Detail |
|---|---|
| RTO target | 5 minutes (via failover to app1-bu) |
| RPO target | 15 minutes (Hermes state), 24h (Docker data) |
| Actual RTO | ~3 minutes (corrected watchdog) |
| Actual RPO | Varies by data type (see below) |
**Restore steps:**
```bash
# Option A: Failover to standby (fastest — < 5 min)
# app1-bu detects Core down → auto-takeover (Section 4.1)
# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on netcup
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
--endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
--endpoint-url https://s3.us-east-1.wasabisys.com
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
--endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
(cd "$dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Check Telegram responds to /ping
```
### 6.2 ai.itpropartner.com (Hetzner CPX41) — 🔴 Critical
| Attribute | Detail |
|---|---|
| RTO target | 2 hours |
| RPO target | 24 hours |
| Current disk | **🔴 92%** — must clean before migration |
**⚠️ Pre-reboot checklist:**
```bash
# Verify at least one alternative model provider works
curl https://admin-ai.itpropartner.com/v1/chat/completions \
-H "Authorization: Bearer $(cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Core is running
curl -s http://127.0.0.1:11434/api/tags || systemctl start ollama
# Notify all Hermes users (Germaine, Anita, Tony)
```
**Restore steps:**
```bash
# Step 1 — Provision replacement (CPX41 or netcup equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: admin-ai.itpropartner.com → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles
```
### 6.3 wphost02 (Hetzner CPX21, RunCloud) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Backup method | RunCloud daily backup + manual DB |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 on Hetzner
# Step 2 — Connect to RunCloud panel, link server
# Step 3 — RunCloud provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from RunCloud backup → verify apex site loads
# Step 5 — Deploy itpp-infra SSH key in RunCloud → SSH Access
# Step 6 — Whitelist Core IP (152.53.192.33) in RunCloud firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:Catches.bullets1985" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f
```
### 6.4 UNMS (Hetzner CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 9 (Docker compose stack) |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS
```
### 6.5 Hudu (Hetzner CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 5 (Docker compose) |
| Auto-backup | Hudu built-in S3 backup |
**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible
```
### 6.6 UniFi (Hetzner CPX21) — 🟡 Important
| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Type | Native install (not Docker) |
**Restore steps:**
```bash
# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches
```
### 6.7 n8n (Hetzner CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 2 (n8n + postgres) |
**Restore:**
```bash
# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d
```
### 6.8 docker box (Hetzner CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 8 (NPM, RustDesk, Uptime Kuma, RunCloud) |
**Restore:**
```bash
# Provision → Docker → restore compose stacks + volumes
```
### 6.9 fleettracker360 (Hetzner CPX11) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Type | Apache + MariaDB (LAMP) |
**Restore:**
```bash
# Provision → Apache + MariaDB → restore web root + DB
```
### 6.10 tony-vps (Hetzner CPX21) — 🟢 Standard
| Attribute | Detail |
|---|---|
| RTO target | Coordinate with Tony |
| Type | Independent Hermes + local LLM |
**⚠️ SHADOW IT — needs documentation.**
**Current known config:**
- Hermes gateway with admin-ai model access
- Llama 3.2 1B local fallback (systemd: llama-server)
- Cron: hermes-heartbeat.sh every minute
- Minimal services, no Docker
**Actions needed:**
1. Document Tony's full Hermes config.yaml and .env
2. Backup to S3 under s3://hermes-vps-backups/tony-vps/
3. Add Tony's config to DR plan once documented
### 6.11 app1-bu (Hetzner CPX21) — 🔴 Standby
| Attribute | Detail |
|---|---|
| RTO target | 2 minutes |
| RPO target | 15 minutes |
| Type | Warm standby — gateway dormant |
**Activation:**
```bash
# Automatic: watchdog detects Core down → S3 sync → start gateway
# Manual:
ssh ippadmin@5.161.114.8
sudo -i
hermes gateway run --replace
```
---
## 7. Testing & Validation
### 7.1 Restore Testing Schedule
| Test | Frequency | Who |
|---|---|---|
| Verify backup job completion (all servers) | Daily | Automated alert |
| Check backup age per server | Daily | Automated |
| Restore one database dump to test environment | Monthly | Network Team |
| Restore one full application stack | Quarterly | Network Team |
| Full DR simulation — Core failover to app1-bu | Quarterly | Network Team |
| S3 credential restore/access test | Quarterly | Network Team |
| Document restore test results (date, tester, duration, findings) | Per test | Network Team |
### 7.2 Validation Criteria
Each restore test must produce:
- Date + tester name
- Server/application tested
- Backup artifact used (path + timestamp)
- Restore duration (start → fully operational)
- Data validation: check a known record exists (specific row, file, or document)
- Issues encountered
- Remediation ticket (if needed)
### 7.3 Quarterly DR Simulation
Scenario: "Core is completely gone. Netcup account inaccessible. app1-bu has been active for 4 hours. Now netcup is back — fail back."
Steps to validate:
1. Simulate netcup provider outage (block Core IP)
2. Verify app1-bu takes over within 5 minutes
3. Send a Telegram message through standby
4. Verify all cron jobs run on standby
5. Simulate netcup recovery
6. Execute manual failback per Section 4.4
7. Verify no data loss, no split-brain
---
## 8. Security
### 8.1 Current Security Posture
| Control | Status | Notes |
|---|---|---|
| SSH keys only | ✅ Done | Password auth disabled |
| Root SSH disabled | ⚠️ Partial | Some servers still allow root |
| UFW host firewall | ✅ All | Standard on all boxes |
| fail2ban | ✅ All | Standard on all boxes |
| S3 versioning | ✅ All buckets | Enabled |
| S3 Object Lock | ❌ Not done | Needs implementation |
| S3 write-only IAM | ❌ Not done | Currently using admin creds |
| Backup encryption | ❌ Not done | Server-side only; no client-side |
| Secrets escrow | 🟡 Partial | .env + Hudu — not all secrets documented |
| TLS everywhere | ✅ All public endpoints | Caddy auto-certs |
### 8.2 Security Gaps to Close
| Priority | Gap | Fix |
|---|---|---|
| 🔴 P0 | S3 uses admin credentials (can delete) | Create IAM sub-user with s3:PutObject only |
| 🔴 P0 | No Object Lock on backups | Enable 7-day minimum retention |
| 🟡 P1 | No client-side backup encryption | Add age/gpg before upload for secondary copy |
| 🟡 P1 | Secrets not fully documented | Audit .env, create secrets inventory in Hudu |
| 🟡 P1 | Some servers allow root SSH | Verify and disable across all boxes |
| 🟢 P2 | No vulnerability scanning | Add lynis weekly scan (already scheduled) |
### 8.3 Break-Glass Access
If itpp-infra key is lost or revoked:
1. **Hetzner servers:** Use Hetzner console → Rescue mode → mount filesystem → redeploy key
2. **netcup servers:** Use SCP web console → VNC → log in with root password → redeploy key
3. **Root passwords:** Stored in ~/.hermes/.env (chmod 600) and Hudu
---
## 9. Immediate Action Items
### 🔴 P0 — This Week
| # | Action | Owner | Details |
|---|---|---|---|
| 1 | Clean up AI server disk | Sho'Nuff | 92% full — purge old Docker images, logs, temp data |
| 2 | Create S3 write-only IAM user | Sho'Nuff | Separate creds from admin — no delete, no list |
| 3 | Enable S3 Object Lock | Sho'Nuff | 7-day minimum on all backup buckets |
| 4 | Deploy volume backup script | Sho'Nuff | All Docker hosts → S3 per-server buckets |
| 5 | Deploy database dump cron | Sho'Nuff | MariaDB on wphost02 + fleettracker360, every 6h |
### 🟡 P1 — Next 2 Weeks
| # | Action | Owner |
|---|---|---|
| 6 | Fix warm standby watchdog interval (30s not 5min) | Sho'Nuff |
| 7 | Document Tony's Hermes config + backup to S3 | Coordinate with Tony |
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) | Sho'Nuff |
| 9 | Extend snapshot retention from 48h to 30 days | Sho'Nuff |
| 10 | Add backup success/failure monitoring to ops portal | Sho'Nuff |
| 11 | Implement priority tagging (Option A) for memory consolidation | Sho'Nuff |
### 🟢 P2 — Next 30 Days
| # | Action | Owner |
|---|---|---|
| 12 | Full DR simulation for Core failover | Network Team |
| 13 | Monthly restore testing schedule begins | Network Team |
| 14 | Deploy Prometheus + Grafana stack | Sho'Nuff |
| 15 | Add disk threshold alerts for all servers | Sho'Nuff |
| 16 | Begin netcup migration | Sho'Nuff |
---
## 10. Appendix: Third-Party Review Responses
### 10.1 Reviewer 1 Findings — Addressed
| Finding | Addressed In |
|---|---|
| Failover timing mismatch (5min check ≠ 2min RTO) | Section 4.1 — Corrected to 30s check |
| Backup compliance gap (standard says daily, not happening) | Section 3.2 — Gap closure plan |
| AI server 92% disk critical | Section 9 — P0 action item #1 |
| Memory consolidation data loss risk | Consolidation proposal (separate doc) |
| Tony's Hermes undocumented | Section 6.10 + P1 action item #7 |
### 10.2 Reviewer 2 Findings — Addressed
| Finding | Addressed In |
|---|---|
| S3 single point of failure | Section 3.3 — Two-region plan |
| Backup frequency vs RPO mismatch | Section 3.1 — Matrix with status |
| Restore procedures not detailed | Section 6 — Full runbooks per server |
| Failback under-defined | Section 4.4 — Protocol with no-auto-failback rule |
| No backup restore testing | Section 7 — Schedule |
| 48h retention too short | P1 item #9 — Extend to 30 days |
| Backup security not described | Section 8.2 |
| Monitoring not fully defined | Section 5 |
| Application backup details vague | Section 2.3 + Section 6 |
| Secrets recovery not documented | Section 2.2 (base install creds) + Section 8.3 |
| Provider/account outage missing | Section 4.5 |
---
*This plan supersedes all previous DR documentation. Review quarterly or after any infrastructure change.*
references/server-dr-plans.md
+453
View File
@@ -0,0 +1,453 @@
# IT Pro Partner — Per-Server Disaster Recovery Plan
**Author:** Sho'Nuff / Network Services Team
**Date:** July 9, 2026
**Version:** 1.0
**Supersedes:** hermes-dr-plan-v2.md (Hermes-specific); server-provisioning-standard-v1.md (base install)
---
## DR Architecture Overview
```
Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
│
SSH/WG ──► Management plane (via itpp-infra key)
│
S3 (Wasabi) ──► Backup plane (all servers)
```
**Shared dependencies:**
- **SSH access:** `itpp-infra` key (Hetzner vault or deployed per-server)
- **DNS:** Cloudflare (primary) / SiteGround (legacy for itpropartner.com)
- **Backup storage:** Wasabi S3 (us-east-1)
- **SMTP relay:** mail.germainebrown.com:2525 (MXroute)
- **Auth/SSO:** Cloudflare Access (DRE portal)
- **Monitoring:** Prometheus node_exporter on each server
- **VPN:** WireGuard for management access
---
## Server 1: Core (netcup RS 2000 G12)
**Hostname:** core
**IP:** 152.53.192.33
**Provider:** netcup (RS 2000 G12, 8C EPYC 9645, 16GB RAM, 512GB NVMe)
**OS:** Debian 13
**Status:** 🟢 LIVE — Primary Hermes host
### Purpose
Main operations server. Runs Hermes, Docker services, ops portal, WireGuard hub,
SMTP relay, local LLM fallback.
### Services Running
| Service | Type | Port | Notes |
|---------|------|------|-------|
| Hermes Agent | systemd | Telegram | Primary agent - deepseek-chat via admin-ai |
| Caddy | systemd | 80/443 | Reverse proxy for all domains |
| WireGuard (wg0) | systemd | 51821/udp | Tunnel to home router (10.77.0.1) |
| MySQL tunnel | autossh | 33060→wphost02:3306 | Apex database access |
| SearXNG | Docker | 8888 | Self-hosted search |
| DocuSeal | Docker | 3000 | Document signing |
| Vaultwarden | Docker | 8080/tailscale | Password manager |
| Ollama | systemd | 11434 | Local LLM fallback (llama3.2:3b) |
| Ops Portal | systemd | 8090 | FastAPI backend |
| node_exporter | systemd | 9100 | Prometheus metrics |
### Docker Containers
- searxng, docuseal, vaultwarden (compass), + others under /root/docker/
### Backup
- **Hermes state:** Every 15 min to s3://hermes-vps-backups/live/ (state.db + config)
- **Full Hermes:** Daily 5 AM to s3://hermes-vps-backups/hermes-full-backup/
- **Memory snapshots:** Every 10 min to s3://hermes-vps-backups/live/memories/
- **Docker volumes:** Manual (not automated)
### Restore Procedure
```bash
# 1. Provision RS 2000 G12 (or better) on netcup
# 2. Install Debian 13, apply base standard (see server-provisioning-standard-v1.md)
# 3. Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# 4. Restore from S3 backup:
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/<latest> /root/restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
tar -xzf /root/restore.tar.gz -C /root/.hermes/
# 5. Restore Docker stacks:
cd /root/docker/searxng && docker compose up -d
cd /root/docker/docuseal && docker compose up -d
cd /root/docker/vaultwarden && docker compose up -d
# 6. Restore Caddy config from S3:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/ \
--endpoint-url https://s3.us-east-1.wasabisys.com
# 7. Restart services:
systemctl restart caddy
systemctl restart hermes
# 8. Verify: ops.itpropartner.com, Teleegram, WG tunnel
```
### Failover
- **app1-bu** (5.161.114.8) is warm standby. Checks Core every 5 min.
- If Core unreachable for 2 min → app1-bu syncs S3 state + starts Hermes gateway.
- Restore Core, then manually fail back.
---
## Server 2: wphost02 (Hetzner CPX21)
**Hostname:** wphost02
**IP:** 5.161.62.38
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12 (RunCloud managed)
**Status:** 🟡 LIVE — Recently reconnected after SSH firewall fix
### Purpose
Apex Predators Track Experience WordPress site. RunCloud-managed.
WPForms registration + notification emails.
### Services Running
- nginx-rc (RunCloud) — 80/443
- MariaDB — 3306
- Postfix — 25 (relay via mail.germainebrown.com:2525)
- httpd — 81 (RunCloud internal)
- RunCloud agent — 34210
- node_exporter — 9100
### Docker Containers
**None.** RunCloud manages PHP directly.
### Backup
- **RunCloud backup:** Daily via RunCloud panel
- **MariaDB dumps:** Not automated externally
- **WordPress files:** Managed by RunCloud
### Restore Procedure
```bash
# 1. Provision CPX21 (or better) on Hetzner
# 2. Install Debian 12
# 3. Connect to RunCloud via their panel
# 4. RunCloud provisions nginx, MariaDB, PHP
# 5. Restore WordPress from RunCloud backup
# 6. Add itpp-infra SSH key in RunCloud → SSH Access
# 7. Whitelist Core IP (152.53.192.33) in RunCloud firewall
# 8. Restore Postfix relay config:
# - relayhost = mail.germainebrown.com:2525
# - SASL auth with email@redacted
# 9. Verify: apextrackexperience.com loads, forms submit
```
### Dependencies
- Core MySQL tunnel connects here (port 33060→3306)
- Apex mail watchdog monitors via SSH
---
## Server 3: ai.itpropartner.com (Hetzner CPX41)
**Hostname:** ai
**IP:** 178.156.167.181
**Provider:** Hetzner (CPX41, 4C/16G/226GB)
**OS:** Debian 12
**Status:** 🟡 LIVE — 🔴 92% disk full (18G free)
### ⚠️ CRITICAL: Disk at 92%. Needs cleanup before migration.
### Purpose
**Critical infrastructure.** Runs LiteLLM proxy that ALL Hermes instances use
for model inference (deepseek-chat, claude-opus). Also runs:
- Open WebUI — AI chat interface
- Ollama — local LLM inference
- Qdrant — vector database
- Caddy — reverse proxy
### Services Running
- LiteLLM — systemd service, port varies (API at admin-ai.itpropartner.com)
- Docker containers: openwebui, litellm, litellm_postgres, caddy, qdrant, ollama
- node_exporter — 9100
### Backup
- **LiteLLM config:** In /root/. hermes/.env and config.yaml (API key: sk-lbh...)
- **Docker volumes:** Not backed up to S3
- **Ollama models:** Would need re-pull (cache only)
### Restore Procedure
```bash
# 1. Provision CPX41 (or better) on netcup as part of migration
# 2. Install Debian 13, Docker
# 3. Deploy LiteLLM Docker stack:
# - From /opt/litellm/docker-compose.yml
# - Restore postgres data for LiteLLM routes
# 4. Deploy Ollama:
# - Pull model: ollama pull llama3.2:3b
# 5. Deploy Open WebUI:
# - From /opt/openwebui/docker-compose.yml
# 6. Deploy Caddy config for admin-ai.itpropartner.com
# 7. Update DNS: admin-ai.itpropartner.com → new IP
# 8. Update Hermes config.yaml on all profiles to point to new IP
# 9. Verify: curl https://admin-ai.itpropartner.com/v1/models
# 10. Decommission old Hetzner server
```
### 🔴 Pre-Reboot Checklist
```bash
# Before ANY reboot or migration of this server:
# 1. Verify at least ONE alternative model provider works:
curl https://admin-ai.itpropartner.com/v1/chat/completions \
-H "Authorization: Bearer sk-lbh..." \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# 2. Ensure local Ollama on Core is running:
curl http://127.0.0.1:11434/api/tags
# 3. Notify all Hermes users (Germaine, Anita, Tony)
```
### Dependencies
- **ALL Hermes instances** depend on this server's LiteLLM proxy
- If this goes down, Hermes falls back to local Ollama (llama3.2:3b on Core)
- **Tony's Hermes** has no fallback — would be offline
---
## Server 4: unms.forefrontwireless.com (Hetzner CPX21)
**Hostname:** unms
**IP:** 5.161.225.131
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — 4d 11h uptime
### Purpose
UISP/UNMS (WISP management) + UCRM (customer management).
Manages tower routers and customer billing for Forefront Wireless.
### Docker Containers (9)
unms-api, unms-device-ws-1, unms-device-ws-2, unms-nginx, unms-rabbitmq,
unms-postgres, unms-siridb, unms-fluentd, ucrm
### Backup
- **Docker volumes:** postgres data, UCRM data
- **UNMS config:** Exportable from UNMS UI
### Restore
```bash
# 1. Provision CPX21 → Reinstall Debian 12 + Docker
# 2. Restore docker-compose stacks from backup
# 3. Restore Postgres volumes
# 4. Update DNS for unms.forefrontwireless.com
# 5. Re-register tower routers in UNMS
```
---
## Server 5: unifi (Hetzner CPX21)
**Hostname:** unifi
**IP:** 178.156.131.57
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — Recently rebuilt (6h uptime)
### Purpose
UniFi Network Controller — manages Ubiquiti network equipment.
### Services
- MariaDB
- UniFi Controller software (native, not Docker)
### Backup
- UniFi backup via controller UI (automatic backups)
- MariaDB database backup
### Restore
```bash
# 1. Provision CPX21 → Install UniFi controller
# 2. Restore UniFi backup from last export
# 3. Update DNS for unifi.itpropartner.com
# 4. Adopt APs/switches back
```
---
## Server 6: hudu.itpropartner.com (Hetzner CPX21)
**Hostname:** hudu
**IP:** 178.156.130.130
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Hudu IT documentation platform — internal knowledge base.
### Docker Containers (5)
hudu2-app-1, hudu2-worker-1, hudu2-db-1, hudu2-redis-1, letsencrypt
### Backup
- **Hudu automatic:** Built-in backup to S3-compatible storage
- **Postgres DB:** Daily dump
### Restore
```bash
# 1. Provision CPX21 → Install Docker
# 2. Deploy Hudu Docker stack
# 3. Restore Postgres from backup
# 4. Update DNS
```
---
## Server 7: fleettracker360 (Hetzner CPX11)
**Hostname:** fleettrack360
**IP:** 178.156.149.32
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Fleet tracking application (BoxPilot Logistics?).
### Services
- Apache2 (80/443)
- MariaDB (3306)
### Backup
- Apache config + web root
- MariaDB database dump
---
## Server 8: docker (Hetzner CPX11)
**Hostname:** docker
**IP:** 178.156.168.35
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — 🔴 Disk 73% used (9.9G free on 38G)
### Purpose
RunCloud staging + utility containers:
- Nginx Proxy Manager
- RustDesk server (hbbr, hbbs)
- Uptime Kuma
- StrongSwan
### Docker Containers (8)
runcloud_php56, runcloud_php81, runcloud_mariadb, nginx-proxy-manager,
hbbr, hbbs, strongswan, uptime-kuma
---
## Server 9: app1 / n8n (Hetzner CPX11)
**Hostname:** n8n
**IP:** 87.99.144.163
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
n8n workflow automation.
### Docker Containers (2)
n8n, n8n-postgres
---
## Server 10: tony-vps (Hetzner CPX21)
**Hostname:** tony
**IP:** 87.99.159.142
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Tony's standalone Hermes agent. Separate from main infrastructure.
### Services
- llama-server (systemd) — Llama 3.2 1B local fallback
- hermes-heartbeat.sh (cron — every minute)
### Notes
- **Tony's independent instance** — not managed from Core
- Uses admin-ai for primary model, local Llama 1B for fallback
- No Docker containers
---
## Server 11: app1-bu (Hetzner CPX11 — Warm Standby)
**Hostname:** app1-bu.itpropartner.com
**IP:** 5.161.114.8
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 STANDBY — Warm, gateway dormant
### Purpose
**Hermes warm standby.** Clones Core's state via S3 sync.
### Services
- SSH (22)
- Tailscale
- node_exporter (9100)
- WireGuard (wg0 — tunnel to home router at 10.77.0.3)
### Failover Procedure
```bash
# Automatic (watchdog on app1-bu):
# 1. Checks Core every 5 min (ping + Hermes health)
# 2. If Core down for 2 min → sync latest S3 state
# 3. Integrity check on state.db (PRAGMA quick_check)
# 4. Start Hermes gateway → Telegram accessible
# 5. Write ACTIVE_OWNER lock to S3
# Manual activation:
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
hermes gateway run --replace
# Or if Core is fenceable:
ssh root@152.53.192.33 "systemctl stop hermes && systemctl mask hermes"
```
### Restore
```bash
# 1. Power on via Hetzner API or console
# 2. Ensure S3 sync runs:
# /root/.hermes/scripts/hermes-standby-sync.sh
# 3. Check integrity:
# sqlite3 /root/.hermes/state.db "PRAGMA integrity_check;"
# 4. If good → start gateway:
# hermes gateway run
# 5. If corrupt → restore from snapshot:
# aws s3 cp s3://hermes-vps-backups/snapshots/<latest> /root/
```
---
## DR Classification Matrix
| Server | Tier | RTO Target | RPO Target | Restore Complexity |
|--------|------|-----------|-----------|-------------------|
| Core | 🔴 Critical | 5 min (failover) | 15 min | Medium (Docker + Hermes) |
| ai (admin-ai) | 🔴 Critical | 2 hours | 24 hours | High (migration) |
| wphost02 | 🟡 Important | 4 hours | 24 hours | Low (RunCloud restore) |
| unms | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
| hudu | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
| unifi | 🟡 Important | 4 hours | 24 hours | Low (native install) |
| n8n | 🟢 Standard | 8 hours | 24 hours | Low (2 containers) |
| docker | 🟢 Standard | 8 hours | 24 hours | Medium (8 containers) |
| fleettracker360 | 🟢 Standard | 8 hours | 24 hours | Low (LAMP stack) |
| tony-vps | 🟢 Standard | Coordinate | 24 hours | Low (minimal) |
| app1-bu | 🔴 Standby | 2 min | 15 min | Low (S3 sync) |
---
## Recovery Bundle Location
Each server's recovery configs are bundled at:
- `/root/.hermes/recovery-bundle-2026-07-05.md` (Core — needs refresh)
- `/root/.hermes/references/server-dr-plans.md` (this document)
- `/root/.hermes/references/server-provisioning-standard-v1.md` (base install)
**S3 backup of this document:**
`s3://hermes-vps-backups/live/config/server-dr-plans.md`
references/server-inventory.md
+23
View File
@@ -0,0 +1,23 @@
# IT Pro Partner — Server Inventory
**Date:** July 10, 2026
**Purpose:** Comprehensive list of all servers, specifications, IP addresses, roles, and migration statuses.
## Inventory Matrix
| Hostname | Provider | Type/Specs | OS | Primary IP | Monthly Cost | Status | Roles | Last Provisioned | SSH Key | Notes |
|---|---|---|---|---|---|---|---|---|---|---|
| **Core** | netcup | RS 2000 G12 (8C/16G/512GB) | Debian 13 | 152.53.192.33 | ~$24/mo | LIVE | Hermes, portal, Docker, monitoring, Vaultwarden, DocuSeal, SearXNG, Ollama, WireGuard hub | July 2026 | itpp-infra | Primary host |
| **app1** | netcup | RS 4000 G12 (12C/32G/1TB) | Debian 13 | 152.53.36.131 | ~$44/mo | LIVE | Open WebUI, LiteLLM, n8n, Ollama | July 2026 | itpp-infra | AI & workflow |
| **app2** | netcup | RS 4000 G12 (12C/32G/1TB) | Debian 13 | 152.53.39.202 | ~$44/mo | LIVE | UNMS/UISP, UniFi Controller, Traccar | July 2026 | itpp-infra | Networking mgmt |
| **app3** | netcup | RS 4000 G12 (12C/32G/1TB) | Debian 13 | 152.53.241.111 | ~$44/mo | LIVE | WordPress panel + all sites | July 2026 | itpp-infra | Web hosting |
| **app1-bu** (Standby) | Hetzner | CPX11 (2C/2G/40GB) | Debian 13 | 5.161.114.8 | ~$5/mo | LIVE | Warm standby, dormant Hermes | July 2026 | itpp-infra | DR Failover node |
| **AI Server** (ai.itpropartner.com) | Hetzner | CPX41 | Debian 12/13 | 178.156.167.181 | ~$30/mo | MIGRATING TO app1 | LiteLLM, OpenWebUI, Ollama, Qdrant, Postgres | Legacy | itpp-infra | Disk at 92%; needs cleanup |
| **docker box** | Hetzner | CPX11 | Debian 12/13 | 178.156.168.35 | ~$5/mo | LIVE / MIGRATING | nginx-proxy-manager, RustDesk, strongSwan, Uptime Kuma, RunCloud (legacy) | Legacy | itpp-infra | |
| **wphost02** | Hetzner | CPX21 (4C/8G/80GB) | Debian 12 | 5.161.62.38 | ~$14/mo | DECOMMISSIONING | 8 WordPress sites, RunCloud, MariaDB | Legacy | itpp-infra | Migrating to app3 |
| **UNMS** | Hetzner | CPX21 (4C/8G/80GB) | Debian 12/13 | 5.161.225.131 | ~$14/mo | MIGRATING TO app2 | UISP/UNMS Docker stack (9 containers) | Legacy | itpp-infra | |
| **UniFi** | Hetzner | CPX21 (4C/8G/80GB) | Debian 12/13 | 178.156.131.57 | ~$14/mo | MIGRATING TO app2 | UniFi Controller 9.5.21 (native) | Legacy | itpp-infra | |
---
*Note: SSH Key `itpp-infra` is stored in the Hetzner vault and managed centrally. Root login is prohibited across all standard deployments.*
references/server-provisioning-standard-public.md
+98
View File
@@ -0,0 +1,98 @@
## 2. Server Standards
### 2.1 Tier Definitions
| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | Host A, Host B, Host C | |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Primary Host | |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | Standby Host | |
| **Legacy** | Variable Secondary Provider | Debian 12 | Existing (migrate on rebuild) | -44/mo |
### 2.2 Base Install — Verified Checklist
```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York
# S2: system update
apt update && apt upgrade -y
# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable
# S4: monitoring
apt install -y prometheus-node-exporter
# S5: standard user + key
adduser [USER-REDACTED] && usermod -aG sudo [USER-REDACTED]
echo "ssh-ed25519 AAA... company-prefix-infra" >> /home/[USER-REDACTED]/.ssh/authorized_keys
# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd
# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
# S8: Root essentials backup (daily, 3 AM UTC)
scp root-essentials-backup.sh [USER-REDACTED]@<new-server>:/root/
echo "0 3 * * * /root/root-essentials-backup.sh 2>&1 | logger -t root-essentials-backup" | crontab -
```
Backs up: configs, scripts, SSH keys, Caddyfile, systemd units, web files, and project documentation (`/root/projects/`). Excludes databases (covered by separate live sync). Uploads to S3 under `s3://<backup-bucket>/<hostname>/`.
### 2.3 S3 Backup Folder Structure
```
s3://<backup-bucket>/
├── <hostname>/ ← Per-server root essentials (daily, 42-100MB)
├── live/ ← State sync (every 15 min)
├── hermes-full-backup/ ← Full Hermes tarballs (daily)
├── wphost02-backup/ ← Legacy web host (one-time)
├── siteground/ ← SFTP site backups
└── root-backup/ ← Core root essentials (legacy path)
```
### 2.4 Application Data Paths — Per Server
Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies
(See Section 6 — Restore Runbooks for per-server details)
---
## 3. Backup Strategy
### Exception: Root SSH on Admin Servers
Standard policy disables root SSH (`PermitRootLogin no`) on all servers. Exception granted for **app1, app2, app3** — these are infrastructure/admin servers where automated provisioning requires root-level access.
Root access is still restricted to:
- Key-only authentication (no passwords)
- Only from the `itpp-infra` key
- `AllowUsers ippadmin root` — both users allowed
- Password authentication and root login with password remain disabled
This preserves the security baseline while allowing Sho'Nuff to automate deployments without sudo workarounds.
### S7: Docker — Standard path `/docker/<service>/`
All Docker services must be deployed under `/docker/<service>/`, not `/opt/` or `/home/`. This is the canonical path per the Docker Service Deployment standard.
```
mkdir -p /docker/<service>
cd /docker/<service>
# docker-compose.yml + .env live here
docker compose up -d
```
references/server-provisioning-standard-review.md
+165
View File
@@ -0,0 +1,165 @@
# ITPP Server Provisioning Standard — Peer Review
**Reviewer:** Hermes Agent (subagent)
**Skill reviewed:** `devops/server-provisioning-standard` (v1.0.0)
**Date:** 2026-07-09
---
## Executive Summary
The standard is well-structured, the ordering is logical, and the scope (OS provisioning → security hardening → base tooling → monitoring → networking → verification) is appropriate. It would work for someone experienced with the ITPP infrastructure, but has several gaps — some critical (commands that will fail in practice), some moderate (missing steps, incomplete coverage), and some cosmetic (documentation polish). Below is a detailed findings table followed by role-specific analyses.
---
## Findings by Section
| # | Section | Severity | Issue | Recommendation |
|---|---------|----------|-------|---------------|
| F1 | §7 — Base Tooling | **High** | `wget` URL `node_exporter-*.linux-amd64.tar.gz` uses a shell glob (`*`) in a URL. `wget` does not expand globs in URLs — it requests the literal asterisk, which 404s. | Use the GitHub API to resolve the latest version, or pin a known version (e.g., `1.8.2`). Example fix: `curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest \| jq -r '.tag_name[1:]'` to extract version, then construct URL. Or pin `1.8.2` and rely on periodic updates. |
| F2 | §11 — Checklist | **High** | Checklist item "AWS CLI installed with S3 credentials" and "`/root/.aws/credentials chmod 600`" refer to S3 credentials, but **no step in the standard creates AWS credentials**. There is no `aws configure`, no credential file write, and no `mkdir -p /root/.aws`. | Add a step after the AWS CLI venv setup to configure credentials (or document that this is done post-provisioning via a secure channel). Also add `mkdir -p /root/.aws` before `chmod`. |
| F3 | §7 — Base Tooling | **High** | `apt install docker-compose-plugin` depends on the Docker apt repository having been added by the convenience script. Recent versions of `get.docker.com` install `docker-compose-plugin` automatically, making this step redundant; but if the script didn't (e.g., older version or proxy issues), the `apt install` will fail because Docker's repo may not be the only source for this package on Debian 13. | After `curl -fsSL https://get.docker.com \| sh`, run `docker compose version` to verify instead of a separate `apt install`. If the plugin is missing, the script `get.docker.com` can be run with `--version` flags, or add `docker-ce-rootless-extras` to the mix. |
| F4 | §6 & §7 — Ordering | **Medium** | No `apt update` is run before the `apt install` commands in sections 6 (Fail2Ban, unattended-upgrades) and 7 (all base packages). On a minimal Debian 13 netcup install, the package cache may be stale or empty, causing `apt install` to fail or install outdated versions. | Add `apt update` at the start of §6 (or at the top of §3 before any package ops). |
| F5 | §5, §6 — SSH config split | **Medium** | SSH password authentication is disabled in §5 (two `sed` commands on sshd_config), then `systemctl restart sshd` runs immediately. In §6, additional SSH hardening (`PermitRootLogin no`, `AllowUsers`, etc.) is listed as "In sshd_config" but there are no follow-up commands to apply those settings. This means the SSH daemon is restarted twice (once in §5, once in §6) with a configuration that's only partially applied after the first restart. | Consolidate all sshd_config settings into a single block in §6, then restart `sshd` once. Use a heredoc or `sed` to apply all settings in one pass, then restart. |
| F6 | §5 — SSH key deploy | **Medium** | `ssh-copy-id -i /root/.ssh/itpp-infra.pub ippadmin@<server-ip>` assumes the command is run **from Core**. If a team member provisions from a different workstation, this path won't exist. Also, there's no instruction on how to obtain the `itpp-infra` private key onto the provisioning machine. | Either (a) document that provisioning must be done from Core, or (b) provide alternative instructions for agents/workstations that have the key loaded into ssh-agent. |
| F7 | §8 — node_exporter | **Medium** | The wildcard in `wget ... node_exporter-*.linux-amd64.tar.gz` also means the `tar` line's glob pattern may not match the actual downloaded filename. And `rm -rf /tmp/node_exporter*` removes files matching `*` under /tmp, but the extractions went to a subdirectory — need to verify the cleanup path. | Combine with F1 fix: use a resolved version string for all three operations (download, extract, clean up). |
| F8 | §9 — Tailscale | **Medium** | `tailscale up --authkey <key>` is documented with a placeholder, but there's no guidance on generating or rotating the auth key. Tailscale auth keys are one-time-use unless marked as reusable; if the key expires mid-provisioning, the step fails silently. | Add a prerequisite step: "In Tailscale admin console, generate an auth key tagged for the `itpp-infra` tag (or appropriate ACL tag) with 'reusable' checked if multiple servers share a key." Also verify with `tailscale status` afterward. |
| F9 | §10 — DNS | **Medium** | "Cloudflare DNS: A record for server IP under itpropartner.com" is too terse for a reproducible procedure. No mention of API token, zone ID, or manual steps. Someone who's never done this will be stuck. | Add either (a) a Cloudflare API curl command to create the A record, or (b) a link to the Cloudflare dashboard with step-by-step instructions. |
| F10 | §9 — WireGuard | **Medium** | "WireGuard configured per-server if direct tunnel to home network needed (10.77.0.0/24)" is mentioned but not implemented. No config file, no `wg-quick` setup, no key generation. | Either remove the statement (if WireGuard is out of scope for base provisioning) or add the full WireGuard setup procedure as a subsection. |
| F11 | §8 — Monitoring | **Low** | "Prometheus server on Core auto-discovers via /opt/prometheus/prometheus.yml" is vague. Auto-discovery implies a file_sd_config, DNS SRV records, or a Consul service — none of which are documented. A new admin won't know how to add the new server to Prometheus. | Add a brief note: "On Core, add the new server's IP and port 9100 to the file_sd target list at `/opt/prometheus/targets/<server-name>.yml` and reload Prometheus with `curl -X POST http://localhost:9090/-/reload`." Or point to the relevant Prometheus config. |
| F12 | §8 — UFW & node_exporter | **Low** | Port 9100 (node_exporter) is opened at the system level but there's no UFW rule to restrict it. By default, node_exporter binds to `0.0.0.0:9100`, so it's accessible to anyone who knows the IP. | Add `ufw allow from <core-ip> to any port 9100 proto tcp` before enabling UFW. Or add `--web.listen-address=127.0.0.1:9100` to the ExecStart line if only local Prometheus scraping is needed. |
| F13 | §10 — Hostname | **Low** | `hostnamectl set-hostname <server-name>` changes the hostname, but `/etc/hosts` should also be updated so the hostname resolves to the server's IP (avoids `sudo` slow-DNS warnings and mailname issues). | Add: `sed -i "s/127.0.1.1.*/127.0.1.1 <server-name>/" /etc/hosts` or a similar fix. |
| F14 | §6 — Unattended upgrades | **Low** | `dpkg-reconfigure -plow unattended-upgrades` is interactive and displays a dialog box. On a headless/minimal Debian without `debconf-utils`, this may hang waiting for user input. | Use `echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true \| debconf-set-selections` before the install, or configure via `/etc/apt/apt.conf.d/20auto-upgrades` directly. |
| F15 | §12 — Documentation | **Low** | §12 says update `itpp-recovery-manual.md` and `app-inventory.csv` after provisioning, but doesn't specify what fields to populate in each. | Add a template or field list: e.g., "Add server name, IP, hostname, purpose, provisioning date to `app-inventory.csv`." Add "Append server's IP, root password location, and SSH key fingerprint to `itpp-recovery-manual.md`." |
| F16 | §4 — ippadmin creation | **Low** | `adduser ippadmin` is interactive (prompts for password, full name, etc.). In an automated/scripted context, this hangs. | Use `adduser --disabled-password --gecos "" ippadmin` and then `passwd ippadmin` separately, or use `useradd -m -s /bin/bash ippadmin` followed by explicit password setting. |
| F17 | §6 — UFW enable | **Low** | `ufw --force enable` can still fail if `ufw allow ssh` wasn't successful (e.g., ssh service name not recognized). Also, UFW may need `rsyslog` to log denials. | Add `ufw logging medium` before enabling, and verify with `ufw status verbose` after enable. This helps debug connectivity issues later. |
| F18 | §7 — NVMe tools on non-NVMe hardware | **Low** | `nvme-cli` is installed as a "system tool". The Standby tier (Hetzner CPX11, 40GB) may use standard SSD, not NVMe. This isn't harmful but is worth noting. | Recommend: either keep it (no-op on non-NVMe) or conditionally install based on storage type. Low-priority suggestion. |
| F19 | §7 — Python section order | **Low** | `python3-pip` requires `python3` (dependency), and `python3-venv` also depends on `python3`. Listing all three explicitly is fine, but there's a subtle issue: on Debian minimal, `python3-pip` may pull in `python3` automatically. Not a bug, just slightly redundant. | No change needed — explicit is better than implicit for a standard. |
| F20 | §2 — Ordering | **Info** | Hetzner CPX11 is listed for the Standby tier, but §2 says "All servers are ordered from **netcup**". Hetzner and netcup are different providers with different ordering processes. | Clarify that the Standby tier is ordered from Hetzner with a separate ordering process (or note that the source of truth for Hetzner provisioning lives elsewhere). |
| F21 | §1 — Hardware Standard | **Info** | The RS 4000 G12 and RS 2000 G12 are netcup product names. If netcup changes their product line, this section will need updates. | Consider adding a netcup product page link so someone can verify current specs. No action needed now. |
| F22 | §3 — Initial Access | **Info** | "servercontrolpanel.de/SCP/" — note this domain is `servercontrolpanel.de`, not a common domain. A new admin might think it's a typo. | Consider adding "(servercontrolpanel.de)" after the mention. No urgent fix. |
---
## Role-Specific Analyses
### Server Admin — Assessment: Good with gaps
**Positive:**
- Ordering from hardware specs → ordering → initial access → user creation → SSH → security → tooling → monitoring → networking → verification is a logical, progressive flow.
- ippadmin with sudo, root-only-for-recovery pattern is sound.
- Docker via official convenience script is the recommended approach.
**Issues to address (from finding table):**
- **F4** — Missing `apt update` before package installs will cause failures on a fresh minimal install.
- **F5** — SSH config split across §5 and §6 with a redundant `sshd restart` means settings may not all land.
- **F14** — Unattended upgrades interactive dialog will hang in non-interactive SSH sessions.
- **F16** — `adduser` interactive prompts will block scripting.
- **F7** — node_exporter download will fail (shell glob in URL).
**Missing base packages to consider:**
- `ca-certificates` (may not be pre-installed on minimal Debian; the Docker convenience script and HTTPS sources need it)
- `gnupg` (needed for apt-key operations if any)
- `systemd-journal-remote` or `logwatch` for centralized log shipping (out of scope but worth mentioning)
- `cron` — is it pre-installed on minimal Debian 13?
**Verdict:** Provisioning order is correct. The standard is missing a few critical commands (`apt update`, F14 fix) and has one major bug (F1/F7). With those fixed, the admin flow is solid.
---
### Network Engineer — Assessment: Needs moderate improvements
**Positive:**
- Tailscale for overlay network is the right choice for a multi-server setup.
- UFW with default-deny and allow-outgoing is correct.
- IPv4 + IPv6 enabled at order time is forward-looking.
- `netcat-openbsd`, `dnsutils`, `traceroute` are the right network diagnostic tools.
**Issues to address:**
- **F8** — Tailscale auth key lifecycle needs documentation.
- **F9** — Cloudflare DNS step is too vague; needs reproducible instructions.
- **F10** — WireGuard is mentioned but not implemented. Either do it or remove it.
- **F12** — node_exporter on 0.0.0.0:9100 is an open port with no firewall restriction.
- **F17** — UFW logging should be enabled for troubleshooting.
- **F20** — Provider mismatch between "all from netcup" and Hetzner standby tier.
**Missing firewall considerations:**
- No ICMP rate limiting or consideration (ping flood protection).
- No mention of UFW application profiles vs raw port rules.
- No `ufw limit ssh` (rate-limiting SSH connections) as an alternative to `ufw allow ssh`.
- No mention of Docker's iptables interference with UFW (Docker manages iptables/nftables rules which can bypass UFW).
**Verdict:** The networking foundation is conceptually correct but lacks enough detail for a junior engineer to execute independently. The Cloudflare DNS step is the weakest link.
---
### Backup Specialist — Assessment: Significant gaps
**Positive:**
- S3 credentials are mentioned (but not implemented — see F2).
- Recovery manual update is noted in §12.
- node_exporter provides CPU/memory/disk metrics for capacity planning.
**Issues to address:**
- **F2** — Critical gap: S3 credentials step is completely missing despite being in the checklist.
- **F11** — No guidance on how to add the new server to Prometheus discovery on Core.
- **F13** — Hostname not in `/etc/hosts` can cause `hostname -f` failures that affect monitoring.
**Missing backup considerations:**
- No backup agent installed (e.g., `restic`, `borg`, `duplicity`, or `aws s3 sync`). While app-specific backups are out of scope for base provisioning, a note about the recovery strategy per server type would help.
- No log rotation policy for node_exporter or systemd journal. Without `journalctl --vacuum-time=7d`, logs grow unbounded on the 40GB standby tier.
- No cron job for nightly `apt update && apt upgrade -y` (the checklist runs it once manually).
- No monitoring threshold for disk space (node_exporter exposes it, but there's no alert rule documented).
- No mention of what data on this server needs backing up at the base OS level (e.g., `/etc`, `/var/lib/docker`, `/opt`).
**Verdict:** The backup specialist's concerns are the least addressed in this standard. The S3 credential gap (F2) is the most impactful — without it, the checklist is untestable and any S3-based backup plan fails.
---
### General & Reproducibility — Assessment: Good skeleton, needs filling
**Positive:**
- The verification checklist is comprehensive (16 items covering SSH, services, tooling, networking, DNS).
- The standard clearly defines what's in scope ("BEFORE any Docker services, databases, or applications").
- Changelog and versioning are present.
- Exceptions process is documented.
**Issues to address:**
- Several commands have placeholders without explanation of where to get the values (`<server-ip>`, `<key>`, `<server-name>`). For most this is fine, but `<key>` in the Tailscale step is opaque.
- The standard assumes the operator is on Core for `ssh-copy-id` but doesn't say so explicitly until the verification phase.
- The Cloudflare DNS step (F9) is the weakest link for a new team member.
- The app-inventory.csv and recovery-manual updates (F15) lack templates or field definitions.
- No rollback instructions — what if a step fails mid-provisioning? A short "If something goes wrong" section would improve confidence.
**Verdict:** A senior team member who knows the ITPP infrastructure can successfully provision a server with this standard. A junior or external contractor needs more detail in several sections.
---
## Recommended Priority Actions
### Must fix before next provisioning (High severity):
1. **F1/F7** — Fix node_exporter download to use a resolved version, not a shell glob in the URL.
2. **F2** — Add AWS credential configuration step (or clearly document it as post-provisioning with a reference).
3. **F4** — Add `apt update` before package installs.
4. **F5** — Consolidate sshd_config changes into one section with one restart.
### Should fix soon (Medium severity):
5. **F8** — Document Tailscale auth key generation process.
6. **F9** — Add Cloudflare DNS creation procedure (API or dashboard).
7. **F10** — Either implement or remove WireGuard reference.
8. **F12** — Restrict node_exporter port with UFW or bind to localhost.
9. **F14** — Make unattended-upgrades config non-interactive.
10. **F16** — Use non-interactive `adduser` flags.
### Nice to have (Low severity):
11. **F13** — Update `/etc/hosts` after hostname change.
12. **F15** — Add field templates for post-provisioning documentation.
13. **F17** — Enable UFW logging.
14. **F11** — Document Prometheus target addition process.
15. **F20** — Clarify Hetzner ordering vs netcup ordering.
---
## Final Verdict
**The standard is conceptually sound and covers the right domains, but has 5 high-severity issues (3 that will cause command failures, 1 missing step, 1 ordering bug) and several medium-severity gaps in reproducibility.** None of the issues are architectural — they're all fixable with targeted edits. With the "must fix" items addressed, this standard is ready for production use. The verification checklist is a strong asset; once the S3 credential configuration is added, every checklist item maps to an actual provisioning step.
references/server-provisioning-standard-v1.md
+137
View File
@@ -0,0 +1,137 @@
# IT Pro Partner — Server Provisioning Standard v1.0
**Author:** Sho'Nuff / Network Services Team
**Date:** July 9, 2026
**Status:** DRAFT — pending review
---
## 1. Server Tier Definitions
| Tier | Spec | OS | Use Case | Monthly Cost |
|------|------|----|----------|-------------|
| **Standard** | RS 4000 G12 (12C EPYC, 32GB RAM, 1TB NVMe) | Debian 13 | app1, app2, app3 | ~$44/mo |
| **Light** | RS 2000 G12 (8C EPYC, 16GB RAM, 512GB NVMe) | Debian 13 | Core (existing) | ~$24/mo |
| **Standby** | CPX11 (2C/2G/40GB) | Debian 12 | app1-bu (warm standby) | ~$8/mo Hetzner |
| **Legacy** | CPX21 (3C/4G/80GB) or CPX41 | Debian 12 | Existing Hetzner boxes | ~$14-44/mo |
---
## 2. OS Provisioning
### OS Selection
- **netcup servers:** Debian 13 (bookworm) — latest stable
- **Hetzner servers:** Debian 12 (existing, migrate to 13 on rebuild)
- **Reasoning:** Debian over Ubuntu for stability, no forced upgrades, smaller footprint
### Base Install
```bash
# Set hostname
hostnamectl set-hostname <server-name>
echo "<server-name>" > /etc/hostname
# Update everything
apt update && apt upgrade -y
# Timezone
timedatectl set-timezone America/New_York
# Fail2ban
apt install -y fail2ban
# Monitoring agent
apt install -y prometheus-node-exporter
# Firewall
apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw --force enable
```
### Standard Users
```bash
# ippadmin — standard admin user for all servers
adduser ippadmin
usermod -aG sudo ippadmin
mkdir -p /home/ippadmin/.ssh
# Deploy itpp-infra SSH key (Hetzner project-wide key)
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" \
>> /home/ippadmin/.ssh/authorized_keys
# Disable password auth for SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
```
### Key Deployment
- **itpp-infra** — standard admin key, deployed to all servers (Hetzner project-wide via SSH vault)
- **Root SSH disabled** after initial setup — use `ippadmin` with `sudo`
---
## 3. Standard Services
Every server gets:
| Service | Purpose | Installed By |
|---------|---------|-------------|
| fail2ban | Brute force protection | Base install |
| prometheus-node-exporter | Metrics for monitoring | Base install |
| ufw | Host firewall | Base install |
| Docker (if needed) | Container runtime | Per-server |
| Caddy (if serving web) | Reverse proxy + TLS | Per-server |
---
## 4. Docker Standard
### Installation
```bash
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
```
### Docker Compose
- All stacks at `/opt/<service>/docker-compose.yml`
- Standard network: bridge (default)
- Volume mounts under `/opt/<service>/data/`
- .env files at `/opt/<service>/.env` (chmod 600)
---
## 5. Backup Standard
| Data | Frequency | Location | Retention |
|------|-----------|----------|-----------|
| System config | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
| Docker volumes | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
| Database dumps | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
| Application data | Per-application | s3://hermes-vps-backups/<server>/ | 90 days |
| Hermes state | Every 15 min | s3://hermes-vps-backups/live/ | 48h (snapshots) |
---
## 6. DR Restore Procedure (Standard)
### Standard Restore
```bash
# 1. Provision new server (same tier or better)
# 2. Run base install steps (Section 2)
# 3. Deploy SSH key
# 4. Restore from S3:
aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ \
--endpoint-url https://s3.us-east-1.wasabisys.com --recursive
# 5. Restore Docker stacks
cd /opt/<service> && docker compose up -d
# 6. Verify health endpoints
```
### Emergency Restore (No S3 Access)
```bash
# Use Hetzner rescue mode / netcup recovery
# Reinstall OS, deploy SSH keys
# Rebuild from documented configs in Hudu
```
references/shark-game-recovery-manual.md
+581
View File
@@ -0,0 +1,581 @@
# 🦈 Shark Attack Fantasy League — Offline Recovery Manual
**Author:** Sho'Nuff | **Audience:** Germaine
**Last updated:** July 9, 2026
**Purpose:** Recover the full shark game system without Sho'Nuff being online.
---
## 1. What Is This? (Quick Overview)
The Shark Attack Fantasy League is a game where players draft shark-attack regions and score points when real shark events hit those regions. It has three parts:
| Part | What it does | Where it lives |
|------|-------------|----------------|
| **Backend** | Python web server + database | `/root/shark-game/backend/` — port 8083 |
| **Frontend** | Web pages you see in the browser | `/var/www/shark-game/` |
| **Scraper** | Fetches shark news and scores points | `/root/shark-game/scraper/` — runs via cron |
**Domains:** `shark.iamgmb.com` (the main site) and `shark.itpropartner.com` (backup URL)
---
## 2. Prerequisites — What You MUST Have Installed
If you're reading this on the server, check these are available:
```
which sqlite3 # Should show /usr/bin/sqlite3
which python3 # Should show /usr/bin/python3
which aws # Should show /opt/awscli-venv/bin/aws
which curl # Should show /usr/bin/curl
which systemctl # Should show /usr/bin/systemctl
which journalctl # Should show /usr/bin/journalctl
```
If any of those are missing, install them:
```bash
# Debian/Ubuntu
apt-get install -y sqlite3 curl systemd
# AWS CLI (if missing)
apt-get install -y python3-pip
pip3 install awscli
```
---
## 3. System Architecture — What Runs Where
```
┌─────────────────────────────────────────────────────────┐
│ Internet │
└──────────┬──────────────────────────────┬────────────────┘
│ │
shark.iamgmb.com shark.itpropartner.com
│ │
▼ ▼
┌──────────────────────────────────────────┐
│ Caddy (reverse proxy) │
│ /etc/caddy/Caddyfile │
│ shark.iamgmb.com → 127.0.0.1:8083 │
│ shark.itpropartner.com → static files │
└──────────────┬───────────────────────────┘
│
┌──────────────▼───────────────────────────┐
│ FastAPI Backend (port 8083) │
│ /root/shark-game/backend/server.py │
│ Runs as: shark-game.service (systemd) │
└──────────────┬───────────────────────────┘
│
┌──────────────▼───────────────────────────┐
│ SQLite Database │
│ /root/shark-game/backend/game.db │
└───────────────────────────────────────────┘
```
**Frontend files** are served by Caddy from `/var/www/shark-game/`:
- `index.html` — Main game page
- `draft-room.html` — Draft interface
- `league.html` — League dashboard
- `settings.html` — User settings
- `how-to-play.html` — Rules
- `help.html` — Help page
- `sw.js` — Service worker (push notifications)
---
## 4. Daily Operations — How to Check the Game
### 4.1 Is the game running?
```bash
systemctl status shark-game
```
**What to look for:**
- **Active: active (running)** ✅ — everything is fine
- **Active: inactive (dead)** ❌ — the service has stopped
- **Active: failed** ❌ — something crashed
### 4.2 View recent logs
```bash
# Last 50 lines
journalctl -u shark-game -n 50
# Follow live logs (like tail -f)
journalctl -u shark-game -n 50 -f
# View today's logs
journalctl -u shark-game --since today
# Filter for errors
journalctl -u shark-game -n 200 | grep -i "error\|traceback\|exception"
```
### 4.3 Check the database
```bash
# Open interactive SQLite shell
sqlite3 /root/shark-game/backend/game.db
# Or run a query directly
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"
# Useful queries:
sqlite3 /root/shark-game/backend/game.db "SELECT id, email, display_name, is_admin FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, name, status, invite_code FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 10;"
```
### 4.4 Check disk space
```bash
df -h /root/shark-game/
```
### 4.5 Restart the game
```bash
systemctl restart shark-game
```
Then verify it came back up:
```bash
sleep 3 && systemctl status shark-game
```
---
## 5. Backup & Restore
### 5.1 How backups work
The game database (`game.db`) is NOT backed up separately — it's included in the **Hermes live-sync** backup, which runs every 15 minutes and uploads to Wasabi S3.
### 5.2 Checking backups exist
```bash
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
```
You should see folders like `live/`, `live-sync/`, `standby/`.
### 5.3 Restoring the database from S3
**Step 1:** Stop the game (so it doesn't write to the DB while restoring)
```bash
systemctl stop shark-game
```
**Step 2:** Download the latest state.db backup
```bash
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
--endpoint-url https://s3.us-east-1.wasabisys.com
```
*(Note: The backup is named `state.db` on S3 but we save it locally as `game.db`)*
**Step 3:** Restart the game
```bash
systemctl start shark-game
```
### 5.4 Restoring the frontend files
The frontend files are in `/var/www/shark-game/`. If they get deleted, you can rebuild them from:
1. **If the git repo exists:** `cd /root/shark-game && git pull`
2. **If the S3 backup has them:** Download them from the `live/` prefix and copy to `/var/www/shark-game/`
3. **If neither exists:** Contact me — I'll need to regenerate the frontend files
### 5.5 Taking a manual backup (for safety)
```bash
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d-%H%M%S)
```
---
## 6. Common Issues & How to Fix Them
### 6.1 "Port 8083 already in use"
**Symptom:** `systemctl start shark-game` fails, or you see "Address already in use" in logs.
**Fix:**
```bash
# Kill whatever is using port 8083, then restart
fuser -k 8083/tcp
systemctl restart shark-game
```
### 6.2 "Database locked" errors
**Symptom:** Users see errors when trying to join leagues, make picks, or view scores. Logs show "database is locked".
**Fix:**
```bash
# Remove the lock file
rm -f /root/shark-game/backend/game.db.lock
rm -f /root/shark-game/backend/game.db-shm
rm -f /root/shark-game/backend/game.db-wal
# Restart the service
systemctl restart shark-game
```
### 6.3 "500 Internal Server Error" when joining a league
**Symptom:** Users can't join leagues. You see "500" errors in the browser.
**Fix:**
```bash
# Check the server logs for details
journalctl -u shark-game -n 50 | grep -A10 "500\|Error\|Traceback"
```
Common causes:
- Missing environment variables (see Section 13)
- Database schema mismatch (after restoring wrong backup)
- Invalid invite code
### 6.4 Push notifications not sending
**Symptom:** Users report they're not getting push notifications.
**Check 1:** Is the service worker loaded?
- Open `https://shark.iamgmb.com/sw.js` in a browser
- You should see JavaScript code, not an error
**Check 2:** Are the VAPID keys set?
```bash
# Check if VAPID keys are in the environment
systemctl show shark-game | grep -i vapid
```
**Check 3:** Test push
```bash
curl -X POST https://shark.iamgmb.com/api/push/test
```
**Fix:** If VAPID keys are missing, restart the service (env vars are loaded at startup):
```bash
systemctl restart shark-game
```
### 6.5 Website shows blank page / 502 Bad Gateway
**Symptom:** `shark.iamgmb.com` shows a blank page or 502 error.
**Fix:**
```bash
# Step 1: Check if backend is running
systemctl status shark-game
# Step 2: Check if Caddy is running
systemctl status caddy
# Step 3: Restart both
systemctl restart shark-game
systemctl restart caddy
```
### 6.6 SSL certificate expired
**Symptom:** Browser shows "Your connection is not private" warning.
**Fix:** Caddy auto-renews certificates. If it didn't:
```bash
systemctl restart caddy
```
---
## 7. Scraper & Scoring
### 7.1 How scoring works
The scraper searches for shark news articles, scores them, and writes pending scores to:
`/root/shark-game/data/pending-scores.json`
The backend picks up these files and awards points to players who drafted the affected regions.
### 7.2 Scheduled runs
The scraper runs automatically via system cron at **8:00 AM ET** every day:
```
0 8 * * * /root/shark-game/scraper/run.sh
```
There's also a **draft reminder** that runs every 15 minutes during draft season.
### 7.3 Manual scrape
If you need to run the scraper right now:
```bash
cd /root/shark-game
source .venv/bin/activate
python3 scraper/scrape.py >> data/scraper.log 2>&1
deactivate
```
Or use the wrapper script:
```bash
cd /root/shark-game && bash scraper/run.sh
```
### 7.4 Check scraper results
```bash
# View the last 50 lines of the scraper log
tail -50 /root/shark-game/data/scraper.log
# Check pending scores
cat /root/shark-game/data/pending-scores.json
```
### 7.5 View the current cron jobs
```bash
crontab -l
```
You should see entries for the scraper and draft reminder.
---
## 8. Push Notifications — How They Work
The game uses **Web Push API** (browser push notifications):
1. **VAPID keys** authenticate the server to send pushes
2. **Service worker** (`/var/www/shark-game/sw.js`) receives pushes in the browser
3. **Users** subscribe via the browser's permission prompt when they visit the site
**VAPID keys** are stored as environment variables and embedded in the server code at `/root/shark-game/backend/server.py`. They include:
- `VAPID_PRIVATE_KEY`
- `VAPID_PUBLIC_KEY`
- `VAPID_CLAIMS` → `mailto:email@redacted`
**Push subscription data** is stored in the `push_subscriptions` table in the database.
### Push notification test
```bash
# Send a test push to all subscribed users
curl -X POST https://shark.iamgmb.com/api/push/test
```
### Check if users are subscribed
```bash
sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;"
```
---
## 9. Quick Command Reference
### Service Management
```bash
systemctl status shark-game # Check if running
systemctl restart shark-game # Restart the backend
systemctl stop shark-game # Stop the backend
systemctl start shark-game # Start the backend
```
### Logs
```bash
journalctl -u shark-game -n 100 # Last 100 log lines
journalctl -u shark-game -n 100 -f # Live tail
journalctl -u shark-game --since today # Today's logs
journalctl -u shark-game -n 200 | grep -i error # Errors only
```
### Database
```bash
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 20;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, display_name, email FROM users WHERE is_admin=1;"
sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;" # Check DB health
```
### Scraper
```bash
cd /root/shark-game && bash scraper/run.sh # Force scrape now
tail -50 /root/shark-game/data/scraper.log # View scraper log
cat /root/shark-game/data/pending-scores.json # View pending scores
crontab -l # List scheduled jobs
```
### Backups
```bash
# Manual backup
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d)
# Check S3 backups
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
# Restore from S3
systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
--endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game
```
### Port / Network
```bash
fuser -k 8083/tcp # Kill anything on port 8083
ss -tlnp | grep 8083 # See what's listening on 8083
curl -I https://shark.iamgmb.com # Test site response
curl http://127.0.0.1:8083/ # Test backend directly
```
### Push Notifications
```bash
curl -X POST https://shark.iamgmb.com/api/push/test # Test push
curl https://shark.iamgmb.com/api/push/vapid-public-key # Check VAPID key
sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;" # Check subs
```
---
## 10. Complete Recovery Flow — If Everything Is Broken
Follow these steps in order if the entire game is down.
### Step 1: Check server is reachable
```bash
ping -c 2 shark.iamgmb.com
```
### Step 2: Check web server (Caddy)
```bash
systemctl status caddy
# If not running: systemctl restart caddy
```
### Step 3: Check backend service
```bash
systemctl status shark-game
# If not running: systemctl start shark-game
```
### Step 4: Check port 8083
```bash
ss -tlnp | grep 8083
# If nothing listening: systemctl restart shark-game
# If wrong process: fuser -k 8083/tcp && systemctl restart shark-game
```
### Step 5: Check the database
```bash
sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;"
# Should return "ok"
```
### Step 6: Check the scraper ran today
```bash
tail -20 /root/shark-game/data/scraper.log
# If last run was yesterday, run it manually
cd /root/shark-game && bash scraper/run.sh
```
### Step 7: Test the site
```bash
curl -I https://shark.iamgmb.com
# Should return HTTP/2 200
```
### Step 8: Restore from backup (if DB is corrupted)
```bash
systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
--endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game
```
---
## 11. File Locations Map
| What | Path |
|------|------|
| Backend server | `/root/shark-game/backend/server.py` |
| Database | `/root/shark-game/backend/game.db` |
| Python virtual environment | `/root/shark-game/backend/venv/` |
| Frontend files | `/var/www/shark-game/` |
| Service worker (push) | `/var/www/shark-game/sw.js` |
| Scraper script | `/root/shark-game/scraper/scrape.py` |
| Scraper wrapper | `/root/shark-game/scraper/run.sh` |
| Scraper dependencies | `/root/shark-game/scraper/requirements.txt` |
| Scraper log | `/root/shark-game/data/scraper.log` |
| Pending scores | `/root/shark-game/data/pending-scores.json` |
| Systemd service | `/etc/systemd/system/shark-game.service` |
| Caddy config | `/etc/caddy/Caddyfile` |
---
## 12. Database Tables Reference
| Table | Purpose |
|-------|---------|
| `users` | Players — email, display name, admin status, notification prefs |
| `leagues` | Leagues — name, invite code, status (draft/active/closed), max players |
| `league_members` | Which users belong to which leagues |
| `league_excluded_regions` | Regions excluded from a league's draft |
| `regions` | Shark attack regions (geo-areas players draft) |
| `draft_picks` | Which player drafted which region in which league |
| `scores` | Score events — what happened, how many points, which region |
| `user_scores` | Running total scores per user |
| `push_subscriptions` | Web push notification subscriptions |
---
## 13. Environment Variables
The game backend needs these environment variables. They're set in the systemd service file or in `/root/.hermes/.env`:
| Variable | Purpose |
|----------|---------|
| `VAPID_PRIVATE_KEY` | Private key for Web Push notifications |
| `VAPID_PUBLIC_KEY` | Public key for Web Push notifications |
To check if they're loaded:
```bash
systemctl show shark-game -P Environment
```
---
## 14. Contact / Escalation
If you've tried everything in this manual and the game is still broken:
1. **Save the logs:**
```bash
journalctl -u shark-game -n 500 > /tmp/shark-game-logs.txt
```
2. **Save the database:**
```bash
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.crash
```
3. **Send me the files.** I can work from logs alone if needed.
---
> **🦈 Remember:** The most common fix is simply `systemctl restart shark-game`. Try that first before doing anything complicated!
references/shark-game-trademark-research.md
+209
View File
@@ -0,0 +1,209 @@
# Trademark Conflict Research Report — "Feeding Frenzy" & Variants
**Date:** July 9, 2026
**Purpose:** Trademark clearance search for a shark-themed fantasy game
**Names Investigated:** "Feeding Frenzy", "Shark Attack Fantasy League", "Shark Attack League"
---
## 1. "FEEDING FRENZY" — USPTO Trademark Summary
### 1.1 Critical Conflict: ELECTRONIC ARTS INC. (LIVE & ENFORCEABLE)
| Field | Detail |
|---|---|
| **Mark** | FEEDING FRENZY |
| **Owner** | Electronic Arts Inc. |
| **Serial #** | 78362441 |
| **Registration #** | 2979806 |
| **Filing Date** | February 4, 2004 |
| **Registration Date** | July 26, 2005 |
| **Status** | **LIVE — REGISTERED AND RENEWED** |
| **Classes** | **Class 9** (game software) — Electronic game programs; downloadable electronic game programs; computer game software; interactive video game software |
| **Next Renewal** | Courtesy reminder sent July 2024 — renewal due by 2025 |
| **Risk** | **🔴 HIGH** |
**Why this matters:** Electronic Arts (EA) owns an active, registered, and renewed federal trademark for "FEEDING FRENZY" covering **exactly the category you'd be filing under** — downloadable computer game software, electronic game programs, interactive video game software. This is the most significant blocker. EA acquired PopCap in 2011, and the Feeding Frenzy games are still sold on Steam.
### 1.2 PopCap Games (CANCELLED — But Relevant for Common Law Rights)
| Field | Detail |
|---|---|
| **Mark** | FEEDING FRENZY |
| **Owner** | PopCap Games, Inc. (now EA) |
| **Serial #** | 77129939 |
| **Registration #** | 3517218 |
| **Filing Date** | March 13, 2007 |
| **Status** | **DEAD — Cancelled Section 8 (2015)** |
| **Classes** | Class 41 (entertainment services — providing websites featuring game content) |
| **Risk** | **🟡 MEDIUM** (common law rights still exist) |
This mark was cancelled in 2015 for failure to file Section 8 renewal. However, PopCap/EA has continued selling Feeding Frenzy games continuously since 2004, establishing strong **common law trademark rights** that are independent of the federal registration.
### 1.3 Other LIVE "FEEDING FRENZY" US Trademarks (Different Goods)
| Owner | Reg. # | Class | Goods | Status | Risk |
|---|---|---|---|---|---|
| Pennington Seed, Inc. | 3341836 | 31 | Wildlife seed mixture (deer food plots) | **LIVE** | 🟢 LOW (different class) |
| Pennington Seed, Inc. (new) | 97908874 | 31 | Wild bird food | **LIVE** (Nov 2024) | 🟢 LOW (different class) |
| Fish Tale Inc. | 2310283 | 25 | Clothing (shirts, T-shirts, sweatshirts) | **LIVE** | 🟢 LOW (different class) |
| San Francisco Bay Brand, Inc. | 2215426 | 16 | Aquarium accessories (fish feeders) | **LIVE** | 🟢 LOW (different class) |
### 1.4 Dead/Cancelled "FEEDING FRENZY" US Trademarks
| Owner | Reg. # | Class | Goods | Status |
|---|---|---|---|---|
| Aristoplay, Ltd. | 1816992 | 28 | Board games/toys | Dead (2001) |
| LivingSocial, Inc. | 4221619 | — | — | Dead |
| Centerline Bait and Tackle | 2419523 | — | Fishing tackles | Dead |
| People Gotta Eat, Inc. | — | — | Snack foods | Abandoned |
| Rac/Rite Distributors | 1052998 | — | — | Dead (since 1976) |
### 1.5 Pending / Other Related Marks
| Mark | Owner | Status | Risk |
|---|---|---|---|
| **MEGA DON: FEEDING FRENZY** | Play'n GO Marks Ltd (slot game) | Live/Pending (filed 2023) | 🟢 LOW (composite mark, different niche) |
| **IGNITE A FEEDING FRENZY** | Hydrowave, LLC | TTAB proceeding | 🟢 LOW (composite mark) |
| **START A FEEDING FRENZY** | Plastic Research & Dev Corp | — | 🟢 LOW (composite mark) |
| **FAMILY FEEDING FRENZY** | Pizza Hut | Dead | 🟢 LOW |
---
## 2. Existing Games Using "Feeding Frenzy" Name
### 2.1 Video Games
| Title | Developer/Publisher | Platform | Year | Still Available? |
|---|---|---|---|---|
| **Feeding Frenzy** | Sprout Games / PopCap Games (now EA) | PC, Xbox 360, Mac, J2ME | 2004 | ✅ Steam (as FF2 Deluxe), but original delisted |
| **Feeding Frenzy 2: Shipwreck Showdown** | Sprout Games / PopCap / EA | PC, Xbox 360 | 2006 | ✅ **On Steam** (as "Feeding Frenzy 2 Deluxe") |
| **Feeding Frenzy (board game)** | Aristoplay | Tabletop | 1992 | ❌ Out of print |
| **Let Me Eat: Feeding Frenzy** | Unclear (clone) | iOS | ~2019 | ✅ App Store (imitation/clone) |
| **Feeding Frenzy - Eat Fish** | Taobien | Android | ~2016 | ✅ Google Play (imitation/clone) |
### 2.2 Feeding Frenzy 2 Deluxe — Steam Page
- **URL:** https://store.steampowered.com/app/3390/
- **Publisher:** Electronic Arts
- **Reviews:** ~2,700 reviews, 97% positive
- **Status:** Active, still for sale
### 2.3 PopCap / EA Franchise
- Official website and game pages still exist on EA's network
- Franchise name "Feeding Frenzy" is actively used in commerce
- EA has strong common law rights from continuous use since 2004
---
## 3. Similar/Confusingly Similar Names
### 3.1 "Shark Frenzy"
| Aspect | Detail |
|---|---|
| **What it is** | Bean bag toss game (SWOOC brand); MakeCode arcade game; TV show "Shark Frenzy: Mating Games" |
| **Trademark** | Likely registered for bean bag game (Class 28) |
| **Risk re: "Feeding Frenzy"** | 🟢 LOW — different mark entirely |
| **Risk re: "Shark Attack League"** | 🟡 MEDIUM — "Shark Frenzy" is a different game product but shares "Shark" theme |
### 3.2 "Fish Frenzy"
| Aspect | Detail |
|---|---|
| **What it is** | Board game (BoardGameGeek); Slot machine ("Gold Fish Frenzy"); Mobile game |
| **Trademark** | "FISH FRENZY" seen on Trademarkia (Larry's Reef Services) |
| **Risk** | 🟢 LOW — different mark |
### 3.3 "Shark Attack Fantasy League"
| Aspect | Detail |
|---|---|
| **Search Results** | No exact match found for this string in any trademark database |
| **Related** | "Shark Attack" amusement park ride (Fantasy Island); "Shark Attack League of BGMI" (esports tournament, India); "Shark League" golf societies |
| **Risk** | 🟢 LOW — No existing trademark conflicts found for the full phrase |
### 3.4 "Shark Attack League"
| Aspect | Detail |
|---|---|
| **Search Results** | No trademark registrations found for this exact phrase |
| **Related** | "Shark Attack League of BGMI" (esports tournament — unregistered) |
| **Risk** | 🟢 LOW — No trademark conflicts found |
---
## 4. Mobile App Store Conflicts
### iOS App Store
- **"Let Me Eat: Feeding Frenzy"** — active app using "Feeding Frenzy" in title
- This is a clone/copycat that likely infringes EA's trademark rights
- Existence proves the App Store has "Feeding Frenzy" as an occupied name space
### Google Play
- **"Feeding Frenzy - Eat Fish"** by Taobien — active app
- **"Aqua Fury: Feeding Fish Frenzy"** — another clone
- Multiple apps using "Feeding Frenzy" in their title
**Risk:** Even though these are likely unauthorized clones, they occupy the app store search results. Any new app with "Feeding Frenzy" in the name would face confusion and likely rejection by Apple/Google if EA enforces its trademark.
---
## 5. Summary of Key Conflicts
### Conflicts by Severity
| # | Name | Source | Similarity | Status | Risk |
|---|---|---|---|---|---|
| 1 | **FEEDING FRENZY** — Electronic Arts | USPTO (Class 9) | **Exact match** | LIVE Registered | 🔴 HIGH |
| 2 | **FEEDING FRENZY** — PopCap/EA common law | Continuous use since 2004 | **Exact match** | Active common law | 🔴 HIGH |
| 3 | **Feeding Frenzy 2 Deluxe** — Steam | Steam store | **Exact match** | Active game for sale | 🔴 HIGH |
| 4 | **MEGA DON: FEEDING FRENZY** — Play'n GO | USPTO (Class 9, 41) | Very close | Pending | 🟡 MEDIUM |
| 5 | **Feeding Frenzy (board game)** — Aristoplay | USPTO (Class 28) | Exact match | Dead (2001) | 🟢 LOW |
| 6 | **FEEDING FRENZY** — Pennington Seed | USPTO (Class 31) | Exact match | LIVE — different class | 🟢 LOW |
| 7 | **Shark Frenzy** | Various | Somewhat similar | Various | 🟢 LOW |
| 8 | **Shark Attack Fantasy League** | No results | Watch | None found | 🟢 LOW |
| 9 | **Shark Attack League** | No results | Watch | None found | 🟢 LOW |
---
## 6. Recommendation
### Is "Feeding Frenzy" safe to trademark?
**NO — "Feeding Frenzy" is NOT safe to trademark for a video game.**
Key reasons:
1. **Electronic Arts owns LIVE Registration #2979806** in Class 9 (computer game software) — the exact class you'd need
2. **EA/PopCap has continuously sold Feeding Frenzy games since 2004** — strong common law rights
3. **The mark is still actively in commerce** — "Feeding Frenzy 2 Deluxe" is on Steam right now with 97% positive reviews
4. Any attempt to register "Feeding Frenzy" for a game would face a near-certain likelihood of confusion refusal from the USPTO examiner, and EA would almost certainly oppose
### Are "Shark Attack Fantasy League" or "Shark Attack League" safe?
**Likely SAFE** — No exact matches found in USPTO, trademark databases, Steam, or app stores. However, a full clearance search by a trademark attorney is recommended before filing.
### Suggested Alternative Names (Preliminary — needs independent verification)
Names that appear potentially available (based on this research):
- **Shark Attack Fantasy League** — clear, descriptive, no current conflicts found
- **Shark Attack League** — shorter variant, seems clear
- **Shark Frenzy** — some existing use as a bean bag toss game (different class) but might be available for a video game — needs deeper checks
- **Feeding Frenzy** is NOT recommended under any circumstances for a video game
### Important Disclaimers
- This research is preliminary and based on publicly available databases
- A comprehensive trademark clearance search by a licensed trademark attorney is strongly recommended before filing
- Common law rights (unregistered trademarks) are harder to detect and may exist for similar names
- State trademark registrations were not searched
- International trademark registrations were not comprehensively searched (only Canadian noted)
- The "Shark Attack" name space has various unregistered uses (esports tournaments, golf leagues, amusement rides) that could theoretically cause confusion depending on the specific game concept
---
## 7. Sources Referenced
- **USPTO via uspto.report:** Serial #78362441 (EA), #78861825 (Pennington), #77129939 (PopCap), #75362047 (Fish Tale), #75310760 (SF Bay Brand), #74392104 (Aristoplay), #97908874 (Pennington new)
- **Justia Trademarks:** https://trademarks.justia.com/771/29/feeding-77129939.html
- **Trademarkia:** https://www.trademarkia.com/feeding-frenzy-74392104
- **Steam:** https://store.steampowered.com/app/3390/Feeding_Frenzy_2_Deluxe/
- **Wikipedia:** https://en.wikipedia.org/wiki/Feeding_Frenzy_(video_game)
- **BoardGameGeek:** https://boardgamegeek.com/boardgame/5098/feeding-frenzy
- **Apple App Store:** "Let Me Eat: Feeding Frenzy"
- **Google Play:** "Feeding Frenzy - Eat Fish" by Taobien
references/sharkbait-domain-research.md
+200
View File
@@ -0,0 +1,200 @@
# Shark Bait / Sharkbait — Domain & Name Availability Research
**Date:** July 9, 2026
**Purpose:** Assess conflicts and availability of "Shark Bait" / "Sharkbait" as a game name and domain
---
## 1. USPTO Trademark Search
### "SHARK BAIT" — Computer Program (Class 9 — Software)
| Field | Value |
|---|---|
| **Serial Number** | 78154729 |
| **Registration Number** | 2729288 |
| **Filed** | Aug 15, 2002 |
| **Registered** | Jun 24, 2003 |
| **Status** | **DEAD — Cancelled** (Jan 31, 2010) |
| **Reason** | Registrant did not file Section 8 declaration |
| **Goods/Services** | Computer program for analyzing real estate related data (NOT games) |
| **Link** | https://tsdr.uspto.gov/#caseNumber=78154729 |
**Verdict: No conflict.** Dead/cancelled, and was never for games/entertainment.
### Other "SHARK BAIT" / "SHARKBAIT" Trademarks (non-game)
| Mark | SN | Goods | Status |
|---|---|---|---|
| SHARK BAIT | 85098948 | Clothing (shirts, hats, swimwear, etc.) | Dead (estimated — no Class 9/41) |
| SHARK BAIT SUPPLY CO. | 87716475 | Clothing | Unknown |
| SHARK BAIT | 75511495 | Decals | Dead |
**No LIVE/ACTIVE trademark for "SHARK BAIT" or "SHARKBAIT" was found in the game-related classes:**
- **Class 9:** Downloadable computer game software, video game software
- **Class 41:** Entertainment services, providing online games
### Trademark Risk Assessment
- **Risk: LOW** — No active game-related trademark for either name
- A new "SHARK BAIT" trademark for a game could be filed with the USPTO
- Existing dead marks in unrelated classes (clothing, real estate) do not pose a conflict
---
## 2. Existing Games Named "Shark Bait" or Similar
### 2a. "Shark Bait" — Mobile Dating Sim (Dorian app)
| Field | Value |
|---|---|
| **Platform** | Dorian app (iOS & Android on Google Play) |
| **Developer** | RoseMagpie (Robin Pendragon) |
| **Type** | Visual novel / dating sim with shark gods |
| **Release** | 2023 |
| **IMDb** | https://www.imdb.com/title/tt35046178/ |
| **TikTok** | @rosemagpiestudios |
| **Similarity** | **Exact match** — "Shark Bait" |
| **Risk** | **HIGH** — A published game already uses this exact name on iOS/Android |
| **Sequel** | "Shark Bait: Riptide" exists (doesn't seem trademarked either) |
### 2b. "Captain Sharkbait: Voyage for Treasure" — Steam
| Field | Value |
|---|---|
| **Platform** | Steam (App ID 4085510) |
| **Developer** | Dylan Gray / Extra Pickle Studios |
| **Type** | 3D platformer |
| **Release** | Nov 4, 2025 |
| **Similarity** | **Close** — includes "Sharkbait" in the FULL title but is actually a character name |
| **Risk** | **MEDIUM** — Uses "Sharkbait" as part of its full name, but the game itself is called "Captain Sharkbait: Voyage for Treasure", not just "Sharkbait" |
### 2c. "SharkBait" — VR Game (Meta Quest)
| Field | Value |
|---|---|
| **Platform** | Meta Quest store |
| **Developer** | Alt Lab VR |
| **Type** | Spearfishing VR |
| **Similarity** | **Exact match** (single word) |
| **Risk** | **MEDIUM** — VR-only, different platform from typical mobile/PC games |
| **Link** | https://www.altlabvr.com/sharkbait |
### 2d. "Shark Bait" — Unit in Clash Mini (Supercell)
| Field | Value |
|---|---|
| **Game** | Clash Mini (mobile) |
| **Developer** | Supercell |
| **Similarity** | **Exact match** — a character/unit name, not the game itself |
| **Risk** | **LOW** — It's a character/unit within a larger game, not a standalone game title |
### 2e. "Sharkbait" — Weapon in Borderlands 4
| Field | Value |
|---|---|
| **Game** | Borderlands 4 |
| **Developer** | Gearbox / 2K Games |
| **Similarity** | **Exact match** — weapon name, not game title |
| **Risk** | **LOW** — It's a named weapon/item, not a game |
### 2f. "Shark Bait" — Board Game (Melissa & Doug)
| Field | Value |
|---|---|
| **Type** | Children's board game (touch-and-feel with plush shark) |
| **Retailer** | Walmart, Amazon |
| **Similarity** | **Exact match** |
| **Risk** | **MEDIUM** — Physical board game, different market from digital games, but same name |
| **Note** | Not found as a registered trademark specifically for this product |
### 2g. "Shark Bait" — Other uses in gaming
| Context | Game | Type | Risk |
|---|---|---|---|
| "Shark Bait" | Raft | In-game item/tool | LOW |
| "Shark Bait" | The Jackbox Party Pack 2 | Achievement name | LOW |
| "Shark Bait" | Gambillion | Achievement name | LOW |
| "Shark Bait" | Snake 3D Adventures | Achievement name | LOW |
| "Shark Bait" | Shark Tale (2004) | Level/chapter name | LOW |
| "Shark Bait" | Persuasive Games Bogost | Flash advergame (2000s) | LOW |
---
## 3. Domain Availability
### Summary Table
| Domain | Available? | Status | Notes |
|---|---|---|---|
| **sharkbait.com** | **NO** ❌ | Taken (GoDaddy) | Apparel brand site; registered 1996, expires Sep 23, 2027 |
| **sharkbaitgame.com** | **YES** ✅ | Available | Best option |
| **sharkbattleague.com** | **YES** ✅ | Available | Odd mashup name; consider if "Shark Bait League" is intended |
| **sharkbaits.com** | **NO** ❌ | Taken (GoDaddy) | Registered May 2021, expires May 19, 2027 — likely parked |
| **playsharkbait.com** | **YES** ✅ | Available | Good secondary option |
### Domain Details
#### sharkbait.com — TAKEN
- **Registrar:** GoDaddy.com, LLC
- **Created:** 1996-09-23
- **Expires:** 2027-09-23
- **Nameservers:** NS41.DOMAINCONTROL.COM, NS42.DOMAINCONTROL.COM
- **Use:** Shark Bait Apparel (clothing brand)
- **Acquisition:** Would likely require purchase from owner; backorder at expiry possible but unlikely to succeed
#### sharkbaitgame.com — AVAILABLE
- **Best option** for a game-specific domain
- Clean, descriptive, memorable
- **Estimated cost:** ~$10-12/year at standard registrars
#### sharkbattleague.com — AVAILABLE
- Only useful if you want "Shark Bait League" branding
- Similarity to "sharkbattleague" is high, which could be confusing
#### sharkbaits.com — TAKEN
- **Created:** 2021-05-19
- **Expires:** 2027-05-19
- **Registrar:** GoDaddy
- **Status:** Parked (likely squatted)
- **Note:** Could attempt to purchase, but the plural form is less desirable for a game
#### playsharkbait.com — AVAILABLE
- Good secondary/redirect option
- **Estimated cost:** ~$10-12/year
---
## 4. Recommendations
### Domain Priority (Grab immediately)
| Priority | Domain | Reason |
|---|---|---|
| **1st** | **sharkbaitgame.com** | Available, descriptive, perfect for a game |
| **2nd** | **playsharkbait.com** | Available, good action-oriented name |
| **3rd** | **sharkbattleague.com** | Only if league/battle branding is relevant |
### Estimated Annual Domain Costs
- New registrations: ~$10-12/year each at standard registrars (Namecheap, Cloudflare, Porkbun)
- Cloudflare is recommended for at-cost pricing (~$8-9/year)
- Avoid GoDaddy for new registrations due to high renewal prices
### Trademark Recommendations
- **No legal obstacles** to using "Shark Bait" or "Sharkbait" as a game name
- The existing "Shark Bait" mobile dating sim (RoseMagpie/Dorian) is the primary conflict — it uses the exact name and is actively marketed
- Recommend strongly considering a more distinctive name to avoid confusion with the existing "Shark Bait" mobile dating sim (which has ~2 years of market presence)
- If proceeding, **file a US trademark application** in Classes 9 and 41 as early as possible
### Risk Summary by Name Variant
| Name | Main Conflict | Risk Level |
|---|---|---|
| **"Shark Bait"** | Mobile dating sim (RoseMagpie/Dorian, 2023) and Melissa & Doug board game | **HIGH** |
| **"Sharkbait"** | VR game (Alt Lab VR, Meta Quest); weapon in Borderlands 4 | **MEDIUM** |
| **"Captain Sharkbait"** | Steam game (2025) | **MEDIUM** |
| **A more distinctive name** | None | **LOW** |
---
## 5. Sources
- USPTO TSDR: https://tsdr.uspto.gov/ (SN 78154729)
- IMDb: https://www.imdb.com/title/tt35046178/ ("Shark Bait" dating sim)
- Steam: https://store.steampowered.com/app/4085510/ ("Captain Sharkbait: Voyage for Treasure")
- Google Play: https://play.google.com/store/apps/details?id=com.dorian.playtogether (Dorian app)
- Alt Lab VR: https://www.altlabvr.com/sharkbait
- Clash Mini Wiki: https://clashmini.fandom.com/wiki/Shark_Bait
- Borderlands Wiki: https://borderlands.fandom.com/wiki/Sharkbait
- WHOIS query via terminal (sharkbait.com, sharkbaitgame.com, sharkbattleague.com, sharkbaits.com, playsharkbait.com)
- Walmart: https://www.walmart.com/ip/Melissa-Doug-Shark-Bait/50389618
references/shonuff-closings.py
+10
View File
@@ -0,0 +1,10 @@
SHONUFF_CLOSINGS = [
"Who's the Master?",
"Kiss my Converse!",
"Am I the meanest?",
"Am I the prettiest?",
"Am I the baddest mofo low down around this town?",
"All right, Leroy. Who's the one-and-only Master?",
"YOU'LL... NEVER... USE... THIS... FOOT... AGAIN!",
"Catches bullets with his teeth?",
]
references/shonuff-image-b64.txt
+1
View File
File diff suppressed because one or more lines are too long
references/shonuff-prompts.md
+16
View File
@@ -0,0 +1,16 @@
# Sho'Nuff — Image Generation Prompts
## Imagen 4.0 Ultra Prompt
Full-body character illustration of Sho'Nuff, a confident muscular black man in a dynamic fighting stance. He wears a sleeveless red vest with large gold shoulder studs, baggy black pants with red stripes, a wide leopard-print belt, black and red sneakers, and red wristbands. Voluminous curly dark hair, wide confident smile with teeth, one hand extended forward. Bold comic book style, vibrant red and black color scheme, plain white background, full body shot.
## Alternative Styles
### Comic / Anime
Sho'Nuff as a fighting game character portrait, dynamic action pose, dramatic lighting, cel-shaded anime style, red and black color palette, intense expression, energy aura.
### Realistic
Photorealistic portrait of Sho'Nuff, confident smirk, red vest with gold trim, curly hair, dramatic studio lighting, shallow depth of field, cinematic composition.
### Minimalist / Logo
Clean vector-style Sho'Nuff face silhouette, bold black lines, red accents, geometric shapes, modern flat design, suitable for branding.
references/shonuff-signature-format.md
+38
View File
@@ -0,0 +1,38 @@
# Sho'Nuff Email Signature — Canonical Format
## As established July 7, 2026
### Red Divider
```html
<hr style="border: none; height: 2px; width: 40px; background: #cc0000; margin: 14px 0 12px 0; border-radius: 2px;">
```
### Signature Block (table layout)
```html
<table cellpadding="0" cellspacing="0" border="0" style="font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; color: #2c2c2c; font-size: 13px; line-height: 1.6;">
<tr>
<td style="padding-right: 14px; vertical-align: middle; width: 80px;">
<img src="https://core.itpropartner.com/shonuff-signature.png" alt="SB" width="72" height="auto" style="border-radius: 50%; display: block; max-width: 72px;">
</td>
<td style="vertical-align: middle;">
<div style="font-size: 15px; font-weight: 700; color: #1a1a1a; letter-spacing: -0.2px;">Sho'Nuff Brown</div>
<div style="font-size: 12px; color: #888888; margin-bottom: 1px;">Germaine's AI Problem Child</div>
<div style="font-size: 12px; color: #aaaaaa; margin-bottom: 6px;">IT Pro Partner</div>
<table cellpadding="0" cellspacing="0" border="0" style="font-size: 12px; color: #999999; line-height: 1.7;">
<tr>
<td style="padding-right: 4px; vertical-align: top; white-space: nowrap; color: #bbbbbb;">@</td>
<td><a href="mailto:email@redacted" style="color: #555555; text-decoration: none; border-bottom: 1px dotted #cccccc;">email@redacted</a></td>
</tr>
</table>
</td>
</tr>
</table>
```
### Notes
- Sho'Nuff sends FROM email@redacted
- Signature badge: https://core.itpropartner.com/shonuff-signature.png (circular JPEG badge)
- Title: Randomly pulled from ~14 options (including "Germaine's AI Problem Child")
- Company: "iAmGMB"
- Email: email@redacted with dotted underline
- Closing quote ABOVE the red divider, signature block BELOW (closing → divider → badge + name/title/company/email)
- For email-only (not web): use embedded base64 JPEG for maximum compatibility
references/shonuff-signature.py
+63
View File
@@ -0,0 +1,63 @@
"""
Sho'Nuff Email Signature — Permanent Reference
Source of truth for titles, closings, and signature format.
DO NOT DELETE or EDIT without the Master's approval.
"""
import random
TITLES = [
"Germaine's AI Ops Engineer",
"Germaine's Baddest AI Mofo Low Down Around This Town",
"Germaine's One-and-Only Digital Master",
"Germaine's Converse-Kicking Assistant",
"Keeper of Germaine's Teeth (Catches Bullets)",
"Germaine's AI Problem Child",
"Germaine's 24/7 Co-Pilot",
"Germaine's Digital Henchman",
"Germaine's Glow Up Coordinator",
"Germaine's Digital Sidekick",
"Germaine's AI Bodyguard",
"Germaine's Chaos Coordinator",
"Germaine's Full-Time Menace",
]
CLOSINGS = [
"Who's the Master?",
"Kiss my Converse!",
"Am I the meanest?",
"Am I the prettiest?",
"Am I the baddest mofo low down around this town?",
"All right, Leroy. Who's the one-and-only Master?",
"YOU'LL... NEVER... USE... THIS... FOOT... AGAIN!",
"Catches bullets with his teeth?",
]
SIGNATURE_BADGE = "https://core.itpropartner.com/shonuff-signature.png"
RED_DIVIDER = '<hr style="border: none; border-top: 2px solid #cc0000; width: 40px; text-align: left; margin: 20px 0; border-radius: 40px;">'
def build_signature_block():
"""Returns the HTML signature block with random title + closing."""
title = random.choice(TITLES)
closing = random.choice(CLOSINGS)
return f"""
{closing}
{RED_DIVIDER}
<table cellpadding="0" cellspacing="0" style="margin: 0;">
<tr>
<td style="vertical-align: middle; padding-right: 12px;">
<img src="{SIGNATURE_BADGE}" width="48" height="48" style="border-radius: 50%;" alt="Sho'Nuff">
</td>
<td style="vertical-align: middle; font-size: 13px; color: #555; line-height: 1.4;">
<span style="font-weight: bold; color: #1a1a2e;">Sho'Nuff</span><br>
{title}<br>
IT Pro Partner<br>
<a href="mailto:email@redacted" style="color: #2563eb; text-decoration: none;">email@redacted</a>
</td>
</tr>
</table>
"""
references/shonuff-titles.py
+15
View File
@@ -0,0 +1,15 @@
SHONUFF_TITLES = [
"Germaine's AI Ops Engineer",
"Germaine's Baddest AI Mofo Low Down Around This Town",
"Germaine's One-and-Only Digital Master",
"Germaine's Converse-Kicking Assistant",
"Keeper of Germaine's Teeth (Catches Bullets)",
"Germaine's AI Problem Child",
"Germaine's 24/7 Co-Pilot",
"Germaine's Digital Henchman",
"Germaine's Glow Up Coordinator",
"Germaine's Digital Sidekick",
"Germaine's AI Bodyguard",
"Germaine's Chaos Coordinator",
"Germaine's Full-Time Menace",
]
references/tony-vps.txt
+12
View File
@@ -0,0 +1,12 @@
# Tony's VPS — Emergency Access Only
Host: 87.99.159.142
User: root
Key: ~/.ssh/itpp-infra
Name: tony
Type: Hetzner CPX21 (3C/4G, 80GB) ✅ Upgraded from CPX11
Location: Ashburn, VA
OS: Ubuntu 24.04
Firewall: UFW — SSH only
# Tony interacts via Telegram (not yet set up)
# Admin-ai API key pending