ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Sync docs, audit artifacts, project notes, and VerdictTank proposal docs

Browse Source 
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs
- disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md
- clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy)
- projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture
- proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review
- docs/super-search/firecrawl-provider-strategy.md
- updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io
- .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
2026-08-26 02:27:28 -04:00
parent 23e9751d38
commit f5175f1ce0
55 changed files with 14669 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/phase-one/findings/sys-c.md
+215
View File
@@ -0,0 +1,215 @@
# Sys-C Findings: Backup & Disaster Recovery Verification
**Auditor:** Sys-C (claude-sonnet-5) | **Engagement:** ITPP Phase One Audit | **Mode:** READ-ONLY
**Scope:** All 27+ backup targets vs live Wasabi S3, restore-test history, RTO/RPO evidence, 3-2-1 compliance.
**Rule applied throughout (Germaine):** "Backed up is not finished until a restore test is confirmed."
This document was completed across two runs (initial discovery + this resume). All findings below are verified against live S3 listings and source docs as of 2026-08-13, not assumed from documentation alone, per brief rule #3.
---
## 1. Restore-Test Coverage (the critical gap)
Only ONE restore-test event exists in ITPP history: **2026-08-10**, documented in `/root/projects/itpp-infrastructure/disaster-recovery/restore-test-log.md`. It tested exactly **2 of the ~34-37 actual backup targets** enumerated in the backup plan (the plan's own header claims "27 targets," see Finding SYSC-06 on that discrepancy).
| Target tested | Verdict | What was verified | Caveat found |
|---|---|---|---|
| Gitea (app2) | PASS | 117 DB tables, 52 repos, 3/3 sampled repos restored with valid git history via `git log`/`git rev-list` | Bare repos missing `refs/` dir (all refs packed) - restore requires manual `mkdir -p refs/heads refs/tags` workaround. Undocumented in the DR runbook until this test. |
| Vaultwarden (app1) | PASS | 29/29 tables match main vs backup DB, 123 ciphers intact, RSA key valid, WAL recovery clean | None - clean pass. |
**Every other backup target (25-35 of them depending on count) has ZERO restore-test evidence.** This includes every Critical and High tier service in the plan's own RTO/RPO table except Gitea: Hermes Agent (full), Traccar, UniFi/UNMS (the "UISP" critical tier), LiteLLM, n8n, Open WebUI, Twenty CRM. It also includes the standby failover path itself (app1-bu) - no evidence a full standby takeover has ever been drilled.
The restore-test log's own recommendations (written by the same automated tester) explicitly say: "Expand coverage - test remaining backup targets... LiteLLM, OpenWebUI, Hudu, Traccar, etc." and "Full-scale DR drill - after individual tests pass, schedule a coordinated full-stack restore to the standby server." Neither has happened as of this audit.
The restore-test log also flagged (2026-08-10, item 4): `core/vaultwarden/` backups stopped 2026-07-28 at 33 KB, stale/misconfigured - consistent with backup-plan.md's own "Stale S3 Paths - Cleanup Queue" listing that path as safe to delete (service migrated to app1). Not a live risk, just confirms the stale-path cleanup queue is accurate.
---
## 2. Per-Target Cross-Reference Table
Verified live against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/` via `aws s3 ls --recursive` on 2026-08-13. "Last S3 object" is the actual most recent object under that prefix, not the backup-plan.md "Last Verified" column (which is stale documentation from 07-28/08-08 and was not trusted per brief rule #3).
### Core (152.53.192.33)
| # | Target | Schedule | Last S3 object (live) | Status | Retention (objects seen) | Restore-test |
|---|---|---|---|---|---|---|
| 1 | Hermes Agent (full) | Daily 1:00 AM | 2026-08-13 01:02 | OK | 72 dailies | NO |
| 2 | Hermes Live Sync | Every 15 min | 2026-08-13 06:00+ (continuous) | OK | rolling, includes cron output | NO |
| 3 | /root Essentials | Daily 3:00 AM | 2026-08-13 03:00 | OK (gaps: 07-11→07-12, 07-21, 07-22, 07-27 missing) | 31 files | NO |
| 4 | Grafana | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
| 5 | Uptime Kuma | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
| 6 | Docker Volumes (raw tars) | Daily 1:30 AM | **2026-08-08** 03:00 (5 days stale as of 08-13) | **STALE** | 55 objects, stopped growing | NO |
| 7 | Prometheus (TSDB snapshot) | Daily 1:30 AM | 2026-08-13 01:30 | OK | only **4** snapshots retained | NO |
| 8 | Auth API | 03:15 (+ dup 04:35 broken job) | 2026-08-13 03:15 | OK | 8 dailies | NO |
| 9 | Timetrex (undocumented - not in backup-plan.md's 27) | ~03:00 daily | 2026-08-13 03:00 | **OK schedule / BROKEN content** - sql.gz only 346 bytes | 2 days seen at this size | NO |
### App1 (152.53.36.131)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 10 | Open WebUI | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
| 11 | LiteLLM | 3:30 AM | 2026-08-13 02:01 (config) | OK | 43 objects | NO |
| 12 | n8n | 2:00 AM | 2026-08-13 02:00 | OK | 26 dailies | NO |
| 13 | MCP Server Configs | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
| 14 | Vaultwarden | 2:30 AM | 2026-08-13 02:30 | OK | 17 objects | **YES - PASS 08-10** |
| 15 | Komodo | 3:45 AM | 2026-08-13 03:45 | OK | 17 objects | NO |
| 16 | DocuSeal | 4:00 AM | 2026-08-13 04:00 | OK | 17 objects | NO |
| 17 | Twenty CRM | 4:15 AM | 2026-08-13 02:01 (files) | OK | 28 objects | NO |
| 18 | Kokoro TTS | N/A (stateless, no backup by design) | N/A | N/A - by design | N/A | N/A |
### App2 (152.53.39.202)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 19 | Hudu | 7:00 AM | 2026-08-13 07:00 | OK | 31 dumps | NO |
| 20 | Gitea | 8:00 AM | 2026-08-13 12:00 UTC | OK | daily dirs, thousands of repo objects | **YES - PASS 08-10 (refs/ caveat)** |
| 21 | UNMS | 6:00 AM (+ intraday auto) | 2026-08-13 06:00 | OK | 8 objects | NO |
| 22 | UniFi | 2:00 AM | 2026-08-13 02:00 | OK | 10 objects | NO |
| 23 | Traccar | 2:30 AM | 2026-08-13 02:30 | OK | 27 dailies | NO |
| 24 | Technitium DNS | 2:45 AM | 2026-08-13 02:45 | OK | 8 objects | NO |
| 25 | Dawarich | 4:00 AM | 2026-08-13 02:30 | OK | 8 objects | NO |
| 26 | RAGFlow | 4:15 AM | 2026-08-13 04:15 | OK | 27 objects | NO |
### App3 (152.53.241.111)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 27 | CloudPanel DB | 3:00 AM | 2026-08-13 03:00 | OK | 27 dailies | NO |
| 28 | MySQL (all DBs) | 3:00 AM | 2026-08-13 03:00 | OK | 221 objects (multi-DB x days) | NO |
| 29 | WordPress Files (per-site tars) | 3:00 AM | 2026-08-13 03:02 (confirmed via direct listing; do not trust naive sort) | OK | 273 objects | NO |
| 30 | Nginx Configs | 3:00 AM | 2026-08-13 03:03 | OK | 26 dailies | NO |
| 31 | Static Sites | 3:00 AM | 2026-08-13 03:02 | OK | 82 objects | NO |
| 32 | WordPress Snapshots (CloudPanel local) | 1AM/1PM | Local disk only, /opt/backup-restore/snapshots/, NOT in S3 | Out of S3 scope - 30-day local retention only, single point of failure if app3 disk dies | local only | NO |
| 33 | Hexclave (Stack Auth) | 3:30 AM | 2026-08-13 03:31 | OK | 6 objects | NO |
| 34 | modelortho.com | 4:30 AM | **2026-08-08** (5 days stale as of 08-13) | **STALE** | only 2 objects ever (site+configs) | NO |
### wphost02 (Hetzner)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 35 | WordPress (7 sites) | 5:00 AM | 2026-08-13 (per-day directories) | OK - 14-day retention confirmed working (DR-018 resolved) | 14 days x sites, 271+ objects | NO |
### Home Router / WISP
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 36 | MikroTik CCR2004 (home gateway) | 6:00 AM | 2026-08-13 06:01 | OK | 30+ dailies since 07-04, config+log pairs | NO |
| 37 | MikroTik CCR (WISP tower) - DR-017 | 6:00 AM (expected) | **ZERO objects under `wisp-backups/configs/tower*`** | **MISSING entirely** | none | NO |
### External / Gaps
| Target | Schedule | Status | Notes |
|---|---|---|---|
| Hetzner Snapshots (API-driven disk snapshots) | Weekly Mon 5:00 AM | **NOT VERIFIED** - backup-plan.md's own "Last Verified" column is blank (" - "); not in S3 so not checkable via `aws s3 ls`; would need Hetzner Cloud API/console access, which is a live-system check outside this audit's read-only S3 tooling. Flag for Phase Two verification. | |
| SiteGround WordPress (non-MainWP sites) - DR-019 | None found | **MISSING** - `siteground/` prefix returns zero objects | Still open |
| app1-bu warm standby (state.db) | Continuous per design, actual: none for state.db | **COLD** - see Finding SYSC-01 | Config.yaml syncs every 10 min; state.db (2.1 GB) last touched 2026-07-15, ~28.7 days stale |
**Tally:** 34 real backup targets confirmed live and enumerable (excluding Kokoro N/A and the two external gaps that have no backup at all) + 1 undocumented (timetrex) + 2 confirmed-missing (WISP tower, SiteGround) + 1 unverifiable-in-scope (Hetzner snapshots) + 1 local-only-not-S3 (CloudPanel WP snapshots).
- **OK (current, correct schedule):** 30
- **STALE (schedule ok historically but no recent object):** 3 (Docker Volumes raw tars - 5 days; modelortho.com - 5 days; timetrex - running but payload broken/near-empty)
- **MISSING (zero backup coverage found):** 2 (WISP tower router configs, SiteGround non-MainWP sites)
- **UNVERIFIABLE with available read-only tooling:** 1 (Hetzner snapshots - needs Hetzner API access, not S3)
- **Restore-test confirmed:** 2 of 34+ (Gitea, Vaultwarden) - **94%+ of backup targets have never had a restore proven to work.**
---
## 3. RTO / RPO - Evidence-Based vs Plan-Stated
backup-plan.md states an aspirational RTO/RPO table by tier. Below is what the evidence actually supports.
| Tier | Services | Plan-stated RPO/RTO | Evidence-based RPO/RTO | Gap |
|---|---|---|---|---|
| Critical | Hermes Agent | RPO ≤1h / RTO ≤4h (assumes live-sync + standby cutover) | RPO: session state ~15 min (live sync confirmed running); full application state via standby is actually **~29 days stale** because state.db is never synced to app1-bu. RTO: **untested** - no full failover drill on record. | Standby is warm for config only, cold for state. Real RTO on a true Core failure is unknown and likely far worse than 4h once state.db has to be rebuilt/accepted-lossy. |
| Critical | Gitea | RPO ≤1h / RTO ≤4h | RPO: actual backup cadence is **once daily (8 AM)**, so real RPO is up to ~24h, not ≤1h. RTO: one restore test exists, DB+3 sample repos restored in ~30s, but that is a partial single-service test on /tmp, not a timed full-stack recovery - and it required an undocumented manual workaround (`refs/` dirs). | Plan's RPO claim of ≤1h is not supported by the actual cron schedule. RTO figure is aspirational, not measured. |
| Critical | Traccar | RPO ≤1h / RTO ≤4h | RPO: daily 2:30 AM dump only → real RPO ~24h. RTO: **never tested.** | Same RPO overstatement; RTO entirely unverified. |
| Critical | UISP (UniFi/UNMS) | RPO ≤1h / RTO ≤4h | RPO: UniFi backs up 2:00 AM daily, UNMS 6:00 AM + intraday auto-backups (best of the critical tier, effectively sub-daily). RTO: **never tested.** | UNMS RPO is reasonably close to plan; UniFi is daily only. Neither has a proven RTO. |
| High | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | RPO 24h / RTO ≤8h | Vaultwarden: RPO 24h matches actual daily cadence, and RTO is the only tier item with real evidence (~2s restore+verify in the 08-10 test, though that is a minimal single-DB test, not a full service standup with docker-compose). The other four: RPO matches (daily), RTO **never tested.** | Plan's RPO is accurate here; RTO is unverified for 4 of 5 services. |
| Medium | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave | RPO 24h / RTO ≤24h | All backups confirmed current daily, consistent with stated RPO. RTO: **never tested for any of them.** | RPO credible, RTO aspirational only. |
| Low | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow | RPO 24h / RTO ≤48h | Backups current daily (except Prometheus only keeps 4 snapshots of retention - a retention risk, not an RPO problem). RTO never tested. MikroTik home gateway confirmed daily; MikroTik tower has **no backup at all**, so its real RTO/RPO is "recovery from scratch," not 48h. | Tower router falls outside even the Low tier's stated objective because it has zero backup. |
**Bottom line:** RPO figures in the plan are mostly defensible for daily-cadence services but overstated (≤1h) for the three daily-only Critical items. RTO figures across every tier are aspirational targets, not measured outcomes - only Gitea and Vaultwarden have ever actually been restored and timed, and both were partial, /tmp-only tests, not full production-equivalent recoveries.
---
## 4. Findings (severity-rated)
### CRITICAL
**SYSC-01 - Warm standby (app1-bu) is cold for application state, contradicting DR docs.**
Evidence: `state.db` on app1-bu (5.161.225.131) last modified 2026-07-15, ~28.7 days stale, 2.1 GB, while `config.yaml` syncs every 10 minutes. Root cause isolated: `/root/.hermes/scripts/hermes-standby-sync.sh` explicitly excludes state.db by design ("Skips massive state databases"), syncing only config/.env/.bashrc/skills/profiles/plugins/cron/references.
Why it matters: DR docs and the backup plan describe app1-bu as ready to take over "if Core goes down," but a failover today would bring up a standby with current configuration and roughly a month-old session/job/cron history. Anyone relying on that standby for continuity of active work, not just infrastructure config, will lose weeks of state silently.
**SYSC-02 - Duplicate/conflicting auth-api-backup cron jobs.**
Evidence: two cron entries invoke `auth-api-backup.sh` - 03:15 (status ok) and 04:35 (status error, exit 1).
Why it matters: Auth API backs all SSO for ITPP per backup-plan.md's own tier notes. A visibly-failing duplicate job is exactly the kind of noise that gets ignored until the good job also breaks and nobody notices because "the cron always shows an error anyway." Leftover from a rename/migration; needs cleanup, and the failing job should be confirmed harmless (not silently corrupting anything) before removal.
**SYSC-03 - 94%+ of backup targets have never had a restore test.**
Evidence: the single restore-test event on record (2026-08-10) covered exactly 2 targets (Gitea, Vaultwarden) out of 34+ live targets identified. Every other Critical and High tier service (Hermes Agent full backup, Traccar, UniFi, UNMS, LiteLLM, n8n, Open WebUI, Twenty CRM) has zero restore-test evidence.
Why it matters: per Germaine's own stated rule, a backup that has never been restored is not a finished backup - it is an assumption. Ransomware, disk failure, or a bad migration could reveal that any of these 32+ untested backups are unusable (as the Gitea test itself discovered a real, previously-unknown restore blocker: missing `refs/` directories). The fact that the one test performed found a real issue is itself evidence that untested backups carry material risk, not theoretical risk.
### HIGH
**SYSC-04 - WISP CCR tower router configs have zero backup coverage (DR-017, still open).**
Evidence: `s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects. The home gateway router at the same prefix pattern (`configs/home/`) IS backed up daily and current through 2026-08-13.
Why it matters: if the tower router fails or is misconfigured, there is no saved configuration to restore from - full manual rebuild from memory/notes, unlike the home gateway which has 30+ days of dailies.
**SYSC-05 - Docker Volumes (raw tars) and modelortho.com backups are stale, not merely slow.**
Evidence: `volumes/` prefix (grafana_data_final, prometheus_data raw tars) last object 2026-08-08, 5 days stale as of audit date, while the DB-level dumps under `core/grafana/` and `core/prometheus/` continue daily and are current. `app3/modelortho/` last object also 2026-08-08, and it has only ever had 2 objects total (site + nginx config), suggesting it may have run once and stopped, or is intentionally infrequent.
Why it matters: this is independent live confirmation of the DR-002 discovery that `docker-volume-sync.sh` was deleted - its function is only partially replaced. The DB-level SQLite/TSDB dumps for Grafana and Prometheus are fine, but the raw volume-level tars (which would matter for a full container rebuild, not just data recovery) have quietly stopped. modelortho.com's near-total absence of backup history (2 objects ever, both 5 days stale) needs its own look - either it's a low-churn static site where infrequent backup is fine, or its cron entry silently stopped after one run.
### MEDIUM
**SYSC-06 - backup-plan.md's own target count ("27") does not match the actual enumerated inventory.**
Evidence: the document's header states "Backup Inventory (27 targets)" but the tables that follow it enumerate roughly 34-37 discrete backup line items across Core/App1/App2/App3/wphost02/Home Router/External, plus at least one undocumented target (timetrex) discovered only via live S3 inspection, not present in the plan's tables at all.
Why it matters: this is a documentation accuracy problem, not an operational one, but it matters for audit trust - if the plan's own headline number is wrong, other "Last Verified" dates in the same document (mostly frozen at 07-28 or 08-08, weeks stale relative to what's actually running) should not be trusted either, which is exactly why this audit verified everything live against S3 rather than the document.
**SYSC-07 - timetrex backup runs but produces a near-empty dump.**
Evidence: `core/timetrex/timetrex-2026-08-13.sql.gz` is 346 bytes, consistent across the days sampled. The companion storage/config tars are tiny but plausible (115 B, 1.5 KB) for a config-only backup, but a 346-byte SQL dump for what should be an application database is very unlikely to be a real, useful backup.
Why it matters: the schedule "succeeding" (no cron error, file lands in S3 daily) is actively misleading - it looks healthy on a dashboard but the data almost certainly isn't recoverable. This is exactly the kind of false-green backup that a restore test would have caught immediately. Root cause not assumed here per brief rule 2 (could be an empty/decommissioned database, a broken mysqldump auth, or an app issue) - needs Phase Two investigation, not remediation.
**SYSC-08 - Docker Volume Sync function (Prometheus/Grafana raw data) claimed-but-not-restore-verified.**
Evidence: `docker-volume-sync.sh` was deleted per DR-002; its function is claimed to be covered by `hermes-backup.sh`, but this audit found the actual raw-volume backups (`volumes/` prefix) are stale since 08-08 (see SYSC-05) and no restore test exists for any Grafana/Prometheus backup, DB-level or volume-level.
Why it matters: the claim of coverage is not supported by live evidence; the safety net here is unverified on two independent axes (currency and restorability).
**SYSC-09 - Prometheus TSDB snapshot retention is unusually shallow (4 objects).**
Evidence: `core/prometheus/` holds only 4 snapshot objects vs 25+ for comparable daily services (Grafana, Uptime Kuma).
Why it matters: if a problem with Prometheus data isn't noticed within roughly 4 days, there may be no earlier snapshot left to recover from. Likely an intentional retention policy given TSDB snapshot size, but worth confirming it's intentional rather than a bug.
**SYSC-10 - CloudPanel WordPress snapshot layer (app3, local-only) has no offsite copy.**
Evidence: `/opt/backup-restore/snapshot.sh` writes to local disk (`/opt/backup-restore/snapshots/`, 30-day retention) only; nothing under this specific mechanism reaches S3 (the separate `app3/wordpress/` S3 backups are a different script/mechanism and are current).
Why it matters: this specific snapshot layer is a single point of failure - if app3's disk fails, these particular snapshots are gone regardless of retention window. The S3-backed `app3/wordpress/` mechanism is a real offsite safety net for the same sites, so overall WordPress exposure on app3 is mitigated, but the local snapshot layer itself provides false comfort if someone assumes "snapshots" means "offsite."
### MEDIUM (carried forward, unchanged from prior run)
**SYSC-11 - DR-019: SiteGround WordPress sites outside MainWP have no S3 backup.** `siteground/` prefix returns zero objects. Still open.
**SYSC-12 - DR-015: service-health-check/apex-mail-watchdog failing on real remote outages.** Still open, unchanged.
### LOW / INFORMATIONAL
**SYSC-13 - sys-b.md vs live check discrepancy on Gitea/Hudu/UNMS/UniFi.**
sys-b.md reported these backups as "silently failing." This audit's live S3 check found all four running on schedule with fresh, current objects through 2026-08-13 (Hudu 07:00, Gitea 08:00/12:00 UTC, UNMS 06:00, UniFi 02:00). Flagging for conductor reconciliation rather than resolving unilaterally - either sys-b observed a transient failure window, checked a stale cache/log rather than live S3, or the issue was fixed between sys-b's check and this one. Recommend the conductor compare exact check timestamps between the two runs before deciding which report is stale.
**SYSC-14 - /root Essentials backup has 3 schedule gaps (07-11→07-12, 07-21, 07-22, 07-27 missing) but is otherwise current.**
Not fatal (the job clearly runs most days and is current through 08-13), but worth a Phase Two look at why specific days were skipped (server reboot, cron collision, disk pressure) rather than assuming a one-off blip.
**SYSC-15 - Hetzner weekly snapshot verification is outside this audit's read-only S3 tooling.**
backup-plan.md lists "Last Verified: - " (blank) for Hetzner Cloud API-driven snapshots. This audit could not verify snapshot existence/currency using `aws s3` tooling because they are not stored in S3. Flag for Phase Two: someone with Hetzner console/API read access should confirm snapshots are actually being taken weekly as claimed.
---
## 5. 3-2-1 Rule Compliance (quick read)
Most services have: (1) live production copy, (2) daily S3 backup at Wasabi (offsite), and for the app1-bu standby (3) a config-level copy on a second provider (Hetzner) - but that third copy is state-incomplete per SYSC-01. Genuine 3-2-1 gaps:
- WISP tower router: only 1 copy (live device config), zero backups (DR-017).
- SiteGround non-MainWP sites: only 1 copy (live host), zero backups (DR-019).
- CloudPanel local WordPress snapshots: 2 copies but both effectively on the same physical host (live + local snapshot dir) until the separate S3-backed `app3/wordpress/` mechanism is counted as the true offsite leg - which it is, so app3 WordPress overall is fine; the local snapshot layer specifically is not.
- Docker volume raw tars: technically offsite but stale 5 days, functionally degrading toward non-compliance if not fixed.
---
## Notes on Methodology / Limitations
- All S3 currency checks used `aws s3 ls --recursive` against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/`, cross-checked with targeted per-prefix listings where a naive lexical sort across mixed filenames (different site/service names sharing a date) produced a misleading "last" result (caught and corrected for `app3/wordpress/`).
- No restore, config change, or live-system alteration was performed by this auditor, per brief rule 1. All restore-test evidence in this report comes from the pre-existing 2026-08-10 log, not from actions taken during this audit.
- Hetzner API-based snapshot verification and any live SSH-based service checks beyond what the prior Sys-C run already completed were not repeated in this resume, consistent with the instruction to finish, not redo, prior discovery.