Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

references/backup-policy.md

@@ -0,0 +1,72 @@
+# IT Pro Partner - Backup and Disaster Recovery Policy
+
+**Effective Date:** July 10, 2026  
+**Document Owner:** IT Operations Manager / MSP Backup Administrator  
+**Scope:** This policy applies to all systems, data, and configurations managed by IT Pro Partner, including internal operations and client environments.
+
+## 1. Overview
+This policy defines the backup strategies, retention schedules, recovery objectives, and testing procedures for all managed environments. Our backup infrastructure relies primarily on off-site cloud storage via Wasabi S3 to ensure data durability, availability, and separation of concerns.
+
+## 2. Classification and Recovery Point Objectives (RPO)
+
+Data is classified into three tiers to align with business continuity requirements:
+
+| Classification | Description | Target RPO | Examples |
+| :--- | :--- | :--- | :--- |
+| **Critical** | Highly dynamic state and session data essential for real-time operations. | 10 - 15 Minutes | AI state databases, memory consolidation, active session configs. |
+| **Important** | Daily production workloads, client websites, and core infrastructure configs. | 24 Hours | WordPress sites, Hermes full instances, router configurations, project documentation (`/root/projects/`). |
+| **Standard** | Static environments, base server images, or ad-hoc archival workloads. | 7 Days / N/A | Hetzner server snapshots, one-time manual node backups (e.g., wphost02). |
+
+## 3. Backup Schedule and Inventory
+
+The following systems are actively protected under automated backup routines:
+
+| Asset / System | Backup Type / Tool | Schedule | Storage Destination |
+| :--- | :--- | :--- | :--- |
+| **Hermes Live Sync** | Incremental (state.db, sessions, configs) via `hermes-live-sync.sh` | Every 15 minutes | Wasabi S3 |
+| **Memory Consolidation** | Pre-prune state via `backup-before-prune` | Every 10 minutes | Wasabi S3 |
+| **Hermes Full Backup** | Full System Archive via `hermes-backup.sh` | Daily at 01:00 UTC | Wasabi S3 |
+| **Project Documentation** | Included in root essentials backup via `root-essentials-backup.sh` | Daily | Wasabi S3 |
+| **Router Configs** | Configuration dumps via `wisp-backup.py` | Daily | Wasabi S3 (`mikrotik-ccr-backups`) |
+| **WordPress Sites (15)**| Incremental & Full via MainWP + WPvivid Pro | Daily, Weekly, Monthly | Wasabi S3 (`ipp-mainwp-s3`) |
+| **Hetzner Servers** | Full VM Snapshots via `snapshot-hetzner.py`| Weekly (Mondays 05:00 UTC) | Hetzner Cloud Native Storage |
+| **wphost02 Node** | One-time full server backup | Ad-hoc / Manual (Jul 10) | Wasabi S3 |
+
+## 4. Retention Policy
+
+Unless specifically overridden by a distinct client contract, all backups adhere to the following baseline retention periods:
+
+- **Daily Backups:** Retained for 7 days.
+- **Weekly Backups:** Retained for 30 days.
+- **Monthly Backups:** Retained for 12 months.
+- **Continuous / Sub-hourly Syncs (Critical Tier):** Retained for 48 hours before rolling up into daily aggregates.
+
+## 5. Storage and Security
+
+### 5.1 Storage Locations
+All primary off-site backups are stored in Wasabi S3 object storage to leverage immutability (where configured) and high durability. Specific buckets are isolated by workload (e.g., `ipp-mainwp-s3` for client sites, `mikrotik-ccr-backups` for infrastructure).
+
+### 5.2 Encryption Standards
+- **At Rest:** Wasabi server-side encryption (SSE) is enabled by default across all backup buckets. MainWP/WPvivid WordPress backups are encrypted prior to upload.
+- **In Transit:** All backup transfers occur exclusively over TLS 1.2+ secure connections.
+- **Client-Side:** Native client-side encryption is not universally implemented on custom shell scripts at this time (see Section 8).
+
+## 6. Verification and Testing
+
+Consistent validation is critical to ensure data integrity and reliable disaster recovery.
+
+- **Automated Verification:** A daily audit watchdog script monitors the success/failure state of scheduled cron jobs, alerting the operations team of missed or failed jobs.
+- **Restore Testing:** A comprehensive, documented restore test of a randomly selected asset (WordPress site, Hermes state, or Router config) is conducted **Monthly**.
+
+## 7. Roles and Responsibilities
+
+- **Backup Owner:** The IT Operations Manager is accountable for the overarching backup strategy, budget, and policy adherence.
+- **Backup Administrator:** Responsible for day-to-day monitoring, script maintenance, provisioning new buckets, and executing the monthly restore tests.
+
+## 8. Acknowledged Gaps and Roadmap
+
+The following known limitations exist within the current architecture and are targeted for future remediation:
+
+1. **Client-Side Encryption:** Some shell-script backups (e.g., Hermes full backups) rely entirely on server-side S3 encryption and TLS. Client-side GPG/AES encryption prior to transit is planned.
+2. **Docker Volumes:** Automated granular backups of individual Docker persistent volumes are not fully standardized across all infrastructure nodes.
+3. **Database Dumps:** Outside of WordPress/MainWP workflows, bare-metal MySQL/PostgreSQL database dumps are not yet uniformly automated.