ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DR docs audit fix (2026-08-08): fix stale dates, path errors, add RPO/RTO, unbacked services, restore testing, rollback procedure, S3 access instructions

Browse Source
This commit is contained in:
root
2026-08-08 17:25:27 -04:00
parent a522d11c35
commit c4c6609a81
3 changed files with 225 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/app2-recovery-runbook.md
+172
View File
@@ -0,0 +1,172 @@
# app2 Recovery Runbook
**Server:** app2 (netcup RS 4000, 8 vCPU/16 GB/512 GB, 152.53.39.202)
**Status:** Production — no warm standby
**Last Updated:** 2026-08-08
---
## 1. Services Hosted (Impact if Down)
| Service | Domain | Criticality | Impact |
|---|---|---|---|
| Gitea | git.itpropartner.com | **High** | All source code repos unreachable. No git push/pull. |
| UISP (UNMS) | unms.forefrontwireless.com | **High** | WISP CCR tower backups, network management offline. |
| Traccar | fleettracker360.com | **High** | Client-facing GPS tracking product down. |
| Technitium DNS | dns1.itpropartner.com | Medium | Authoritative DNS for internal zones. All ITPP servers use Tailscale MagicDNS (100.100.100.100) for resolution, so internal DNS is NOT affected. Only external clients querying zones hosted on Technitium would be impacted. ⚠ No backup. |
| Hudu | hudu.itpropartner.com | Medium | IT documentation unavailable. Important but not blocking. |
| UniFi Controller | unifi.itpropartner.com | Medium | Wi-Fi management offline. APs continue operating in standalone mode. |
| Dawarich | timeline.iamgmb.com | Low | Personal location tracking. ⚠ No backup. |
| RAGFlow | ragflow.itpropartner.com | Low | RAG pipeline. ⚠ No backup. |
---
## 2. Pre-Flight Checks (Before Recovery)
Before assuming app2 is truly down, verify from Core:
```bash
# 1. Ping check
ping -c 3 -W 2 152.53.39.202
# 2. SSH check (with timeout to avoid hanging)
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'uptime'
# 3. Docker health
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'docker ps --format "{{.Names}} {{.Status}}" | grep -v "Up"'
```
---
## 3. Recovery Scenarios
### Scenario A: app2 is reachable but Docker services are down
```bash
# SSH in, check what happened
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202
# Check disk space (common cause)
df -h
# Check Docker status
systemctl status docker
docker ps -a
# Restart critical services first
docker restart gitea hudu-app-1 traccar
docker restart technitium
```
### Scenario B: app2 is completely unreachable (server crash/hung)
1. **Log into netcup CCP** (Server Control Panel) at https://www.servercontrolpanel.de
2. **Check server status** — if hung, send ACPI shutdown + cold boot
3. **If boot fails:** Request KVM console from netcup support (or use integrated KVM if available)
4. **Boot into rescue mode if needed**, check filesystem:
```bash
fsck -f /dev/vda4
mount /dev/vda4 /mnt
# Check logs
cat /mnt/var/log/syslog | tail -100
```
### Scenario C: Complete server failure (hardware, unrecoverable)
1. **Order new RS 4000 from netcup** — provision with same specs (8 vCPU, 16 GB RAM, 512 GB SSD). Netcup provisioning typically takes **2-4 hours** for existing customers.
2. **Access S3 backups** to download restore data:
```bash
# Credentials are on Core at /root/.aws/credentials (wasabi profile)
# Or use the awscli venv:
source /opt/awscli-venv/bin/activate
# List available backups
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/app2/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/gitea/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/hudu/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unms/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unifi/
# Download latest backup for each service
aws s3 sync --endpoint-url https://s3.us-east-1.wasabisys.com \
s3://hermes-vps-backups/app2/ ./restore/app2/
```
3. **Re-deploy Docker services** using compose files from restored data
4. **Restore databases** from S3 dumps
5. **Update DNS** for app2's new IP (if netcup assigns a different one)
6. **Reinstall Tailscale** and re-approve in admin console
7. **Note:** Netcup typically assigns IPs from the same subnet on re-provision, but this is not guaranteed.
---
## 4. Service-Specific Recovery
### Gitea
- Data: Docker volume at `/var/lib/docker/volumes/gitea_data`
- DB: **SQLite** (`gitea.db`) — file-based, no separate database container. Backup copies the SQLite file directly.
- Restore: `docker restart gitea` is usually sufficient; if data is corrupted, restore `gitea.db` from S3 and restart.
- Backup exists at `s3://hermes-vps-backups/gitea/daily/`
### UISP (UNMS)
- Stack: 9 containers (postgres, siridb, rabbitmq, fluentd, nginx, netflow, api, device-ws x8)
- Data: PostgreSQL at `unms-postgres`, SiridB at `unms-siridb`
- Restore: `cd /root/unms && docker compose up -d`
- Backup: `s3://hermes-vps-backups/unms/`
### Traccar
- Data: H2 database (embedded, no separate DB container)
- Config: XML at `/opt/traccar/conf/traccar.xml`
- Restore: `docker restart traccar`
- Backup: `s3://hermes-vps-backups/app2/traccar/`
### Technitium DNS
- Data: Docker bind mount at `/root/docker/technitium/data` → `/etc/dns`
- **⚠ No S3 backup** — zones exist only on disk. On full server loss, zones must be recreated manually.
- Restore: `docker restart technitium` (zones auto-load from volume on restart)
- Mitigation: Export zone files manually and commit to git for DR coverage until automated backup is implemented.
### Dawarich
- Stack: 4 containers (`dawarich_app`, `dawarich_sidekiq`, `dawarich_db`, `dawarich_redis`)
- **⚠ No backup** — location history data has zero DR coverage.
- Database: PostgreSQL (container `dawarich_db`). To create a backup: `docker exec dawarich_db pg_dump -U postgres dawarich > dawarich.sql`
- Restore: `cd /root/dawarich && docker compose up -d`
### RAGFlow
- Stack: 1 container (`docker-ragflow-cpu-1`)
- **⚠ No backup** — RAG pipeline state has zero DR coverage.
- Data: Docker volume; database engine TBD (needs inspection)
- Restore: `cd /root/ragflow && docker compose up -d`
### Hudu
- Stack: app + db + redis + worker
- Data: PostgreSQL at `hudu-db-1`
- Restore: `cd /root/hudu && docker compose up -d`
- Backup: `s3://hermes-vps-backups/hudu/`
---
## 5. Rollback Procedure
If a recovery attempt makes things worse (wrong backup restored, config mismatch, cascading failures):
1. **Stop affected containers:** `docker stop <container>`
2. **Identify the last known-good backup** from S3 timestamps
3. **Restore from the previous day's backup** (never overwrite the good backup while troubleshooting)
4. **Start containers one at a time** — verify each before starting the next
5. **If services still fail:** Do NOT attempt additional restores. Escalate to Germaine and document what was tried.
> **Golden rule:** The backup you're about to overwrite is your safety net. Copy it aside before restoring over it.
---
## 6. Verification Checklist (Post-Recovery)
- [ ] SSH to app2 works
- [ ] All Docker containers show "Up" in `docker ps`
- [ ] `git clone git.itpropartner.com/ippadmin/itpp-infrastructure.git` succeeds
- [ ] `https://git.itpropartner.com` loads in browser
- [ ] `https://hudu.itpropartner.com` loads in browser
- [ ] `https://unms.forefrontwireless.com` loads in browser
- [ ] `https://unifi.itpropartner.com` loads in browser
- [ ] `https://fleettracker360.com` loads in browser
- [ ] `dig @152.53.39.202 itpropartner.com` returns authoritative answer (DNS)