ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a51db27080a98c937389867c6abcff1654911ead
itpp-infrastructure/docs/app2-recovery-runbook.md
T

7.2 KiB
Raw Blame History

app2 Recovery Runbook

Server: app2 (netcup RS 4000, 8 vCPU/16 GB/512 GB, 152.53.39.202) Status: Production — no warm standby Last Updated: 2026-08-08

1. Services Hosted (Impact if Down)

Service Domain Criticality Impact
Gitea git.itpropartner.com High All source code repos unreachable. No git push/pull.
UISP (UNMS) unms.forefrontwireless.com High WISP CCR tower backups, network management offline.
Traccar fleettracker360.com High Client-facing GPS tracking product down.
Technitium DNS dns1.itpropartner.com Medium Authoritative DNS for internal zones. All ITPP servers use Tailscale MagicDNS (100.100.100.100) for resolution, so internal DNS is NOT affected. Only external clients querying zones hosted on Technitium would be impacted. ⚠ No backup.
Hudu hudu.itpropartner.com Medium IT documentation unavailable. Important but not blocking.
UniFi Controller unifi.itpropartner.com Medium Wi-Fi management offline. APs continue operating in standalone mode.
Dawarich timeline.iamgmb.com Low Personal location tracking. ⚠ No backup.
RAGFlow ragflow.itpropartner.com Low RAG pipeline. ⚠ No backup.

2. Pre-Flight Checks (Before Recovery)

Before assuming app2 is truly down, verify from Core:

# 1. Ping check
ping -c 3 -W 2 152.53.39.202

# 2. SSH check (with timeout to avoid hanging)
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'uptime'

# 3. Docker health
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'docker ps --format "{{.Names}} {{.Status}}" | grep -v "Up"'

3. Recovery Scenarios

Scenario A: app2 is reachable but Docker services are down

# SSH in, check what happened
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202

# Check disk space (common cause)
df -h

# Check Docker status
systemctl status docker
docker ps -a

# Restart critical services first
docker restart gitea hudu-app-1 traccar
docker restart technitium

Scenario B: app2 is completely unreachable (server crash/hung)

  1. Log into netcup CCP (Server Control Panel) at https://www.servercontrolpanel.de
  2. Check server status — if hung, send ACPI shutdown + cold boot
  3. If boot fails: Request KVM console from netcup support (or use integrated KVM if available)
  4. Boot into rescue mode if needed, check filesystem: 
    fsck -f /dev/vda4
mount /dev/vda4 /mnt
# Check logs
cat /mnt/var/log/syslog | tail -100

Scenario C: Complete server failure (hardware, unrecoverable)

  1. Order new RS 4000 from netcup — provision with same specs (8 vCPU, 16 GB RAM, 512 GB SSD). Netcup provisioning typically takes 2-4 hours for existing customers.
  2. Access S3 backups to download restore data: 
    # Credentials are on Core at /root/.aws/credentials (wasabi profile)
# Or use the awscli venv:
source /opt/awscli-venv/bin/activate

# List available backups
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/app2/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/gitea/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/hudu/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unms/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unifi/

# Download latest backup for each service
aws s3 sync --endpoint-url https://s3.us-east-1.wasabisys.com \
  s3://hermes-vps-backups/app2/ ./restore/app2/
  3. Re-deploy Docker services using compose files from restored data
  4. Restore databases from S3 dumps
  5. Update DNS for app2's new IP (if netcup assigns a different one)
  6. Reinstall Tailscale and re-approve in admin console
  7. Note: Netcup typically assigns IPs from the same subnet on re-provision, but this is not guaranteed.

4. Service-Specific Recovery

Gitea

  • Data: Docker volume at /var/lib/docker/volumes/gitea_data
  • DB: SQLite (gitea.db) — file-based, no separate database container. Backup copies the SQLite file directly.
  • Restore: docker restart gitea is usually sufficient; if data is corrupted, restore gitea.db from S3 and restart.
  • Backup exists at s3://hermes-vps-backups/gitea/daily/

UISP (UNMS)

  • Stack: 9 containers (postgres, siridb, rabbitmq, fluentd, nginx, netflow, api, device-ws x8)
  • Data: PostgreSQL at unms-postgres, SiridB at unms-siridb
  • Restore: cd /root/unms && docker compose up -d
  • Backup: s3://hermes-vps-backups/unms/

Traccar

  • Data: H2 database (embedded, no separate DB container)
  • Config: XML at /opt/traccar/conf/traccar.xml
  • Restore: docker restart traccar
  • Backup: s3://hermes-vps-backups/app2/traccar/

Technitium DNS

  • Data: Docker bind mount at /root/docker/technitium/data → /etc/dns
  • ⚠ No S3 backup — zones exist only on disk. On full server loss, zones must be recreated manually.
  • Restore: docker restart technitium (zones auto-load from volume on restart)
  • Mitigation: Export zone files manually and commit to git for DR coverage until automated backup is implemented.

Dawarich

  • Stack: 4 containers (dawarich_app, dawarich_sidekiq, dawarich_db, dawarich_redis)
  • ⚠ No backup — location history data has zero DR coverage.
  • Database: PostgreSQL (container dawarich_db). To create a backup: docker exec dawarich_db pg_dump -U postgres dawarich > dawarich.sql
  • Restore: cd /root/dawarich && docker compose up -d

RAGFlow

  • Stack: 1 container (docker-ragflow-cpu-1)
  • ⚠ No backup — RAG pipeline state has zero DR coverage.
  • Data: Docker volume; database engine TBD (needs inspection)
  • Restore: cd /root/ragflow && docker compose up -d

Hudu

  • Stack: app + db + redis + worker
  • Data: PostgreSQL at hudu-db-1
  • Restore: cd /root/hudu && docker compose up -d
  • Backup: s3://hermes-vps-backups/hudu/

5. Rollback Procedure

If a recovery attempt makes things worse (wrong backup restored, config mismatch, cascading failures):

  1. Stop affected containers: docker stop <container>
  2. Identify the last known-good backup from S3 timestamps
  3. Restore from the previous day's backup (never overwrite the good backup while troubleshooting)
  4. Start containers one at a time — verify each before starting the next
  5. If services still fail: Do NOT attempt additional restores. Escalate to Germaine and document what was tried.

Golden rule: The backup you're about to overwrite is your safety net. Copy it aside before restoring over it.

6. Verification Checklist (Post-Recovery)

  • SSH to app2 works
  • All Docker containers show "Up" in docker ps
  • git clone git.itpropartner.com/ippadmin/itpp-infrastructure.git succeeds
  • https://git.itpropartner.com loads in browser
  • https://hudu.itpropartner.com loads in browser
  • https://unms.forefrontwireless.com loads in browser
  • https://unifi.itpropartner.com loads in browser
  • https://fleettracker360.com loads in browser
  • dig @152.53.39.202 itpropartner.com returns authoritative answer (DNS)
Reference in New Issue View Git Blame Copy Permalink