Jul 17: WP migration complete, Mattermost deployed, backup restore system, DNS updates

2026-07-17 11:39:47 -04:00
parent d2643a5e7c
commit a242be9324
6 changed files with 117 additions and 11 deletions
@@ -0,0 +1,6 @@
 # itpp-infrastructure — CHANGELOG
 ## 2026-07-16 — Audit Remediation
 - Created CHANGELOG.md (missing per project documentation standard)
 - Project directory: `/root/projects/itpp-infrastructure`
@@ -1,6 +1,6 @@
 # IT Pro Partner — Infrastructure Documentation
-> **Last Updated:** July 16, 2026
+> **Last Updated:** July 17, 2026
 > **Maintainer:** Sho'Nuff
 ---
@@ -42,8 +42,10 @@
  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
  - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
  - Caddy (systemd, 80/443)
  - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
  - Super Search MCP (systemd, port 8899)
 ### App2 Server
 - **Hostname:** app2
@@ -65,12 +67,16 @@
 - **Provider:** netcup RS 4000 G12
 - **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
 - **OS:** Debian 13
- **Role:** Web Hosting
+- **Role:** Web Hosting + Backup Restore
 - **Key Services:**
  - CloudPanel CE — panel.itpropartner.com
  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
-  - voipsimplicity.com (WordPress, migrated Jul 16)
+  - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
-  - my.voipsimplicity.com (static HTML SPA, VoIP customer portal)
+  - WordPress sites (7 migrated from wphost02, all live):
    - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
    - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
    - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
  - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
 ### Core-BU (Warm Standby)
 - **Hostname:** core-bu
@@ -249,3 +255,5 @@ All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-rene
 - **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
 - **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306.
 - **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
 - **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
 - **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.
@@ -19,6 +19,7 @@
 | 2:30 AM | app2 | /root/backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi |
 | 3:00 AM | Core | root-essentials-backup.sh | /root essentials |
 | 3:00 AM | app3 | /root/backup.sh | CloudPanel, MySQL, WordPress, Nginx configs |
 | 1:00 AM + 1:00 PM | app3 | /opt/backup-restore/snapshot.sh | Per-site WordPress snapshots (files + DB), 60-day retention |
 | 3:00 AM | Core | docker-volume-sync.sh | Docker volumes |
 | 4:00 AM | Core | system-config-sync.sh | System configs |
 | 6:00 AM | Core | home-router-backup.sh | MikroTik CCR config |
@@ -51,8 +52,9 @@
 | LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
 | Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
 | n8n | Postgres pg_dump → S3 (daily) | 24 hr |
-| MCP servers | Config files → S3 (daily) | 24 hr |
+| Ollama | Models preserved in Docker volume | 24 hr |
-| Ollama | Model list → S3 (daily) | 24 hr |
+| MCP servers | Configs in Git + Docker volumes | 24 hr |
 | Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr |
 ### app2 (152.53.39.202)
@@ -72,6 +74,8 @@
 | MySQL | All databases mysqldump → S3 (daily) | 24 hr |
 | WordPress | Files + wp-config → S3 (daily) | 24 hr |
 | Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr |
 | **WordPress Snapshots** | **Per-site tarball + MySQL dump (2x daily, 60-day retention)** | **12 hr** |
 | **Backup Restore UI** | **Flask app port 8090 at my.itpropartner.com/backups** | **Instant** |
 | voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr |
 | my.voipsimplicity.com | Covered by Git (static HTML) | N/A |
@@ -1,13 +1,12 @@
 # DNS Records — itpropartner.com
 **Managed by:** SiteGround Site Tools (no API access)
-**Updated:** July 16, 2026
+**Updated:** July 17, 2026
 **⚠️ Germaine must apply changes manually at SiteGround**
 ## CREATE (pending)
 | Subdomain | Record | IP | Service |
 |---|---|---|---|
 | my.itpropartner.com | A | 152.53.192.33 | Customer portal hub |
 | vault.itpropartner.com | A | 152.53.36.131 | Vaultwarden (moving from Core) |
 | status.itpropartner.com | A | 152.53.192.33 | Public status page |
@@ -23,18 +22,21 @@
 |---|---|---|---|
 | ops.itpropartner.com | 152.53.192.33 | Core | Operations dashboard |
 | core.itpropartner.com | 152.53.192.33 | Core | Landing page + MikroTik |
 | my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
 | sign.itpropartner.com | 152.53.192.33 | Core | Document signing |
 | uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
 | app.itpropartner.com | 152.53.192.33 | Core | App entry |
 | admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
 | ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
 | n8n.itpropartner.com | 152.53.36.131 | app1 | Workflow automation |
 | **noc.itpropartner.com** | **152.53.36.131** | **app1** | **Mattermost NOC chat** |
 | app1.itpropartner.com | 152.53.36.131 | app1 | App server 1 |
 | git.itpropartner.com | 152.53.39.202 | app2 | Gitea |
 | hudu.itpropartner.com | 152.53.39.202 | app2 | Documentation |
 | unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
 | www.unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi (www) |
 | panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel |
 | mainwp.itpropartner.com | 152.53.241.111 | app3 | MainWP management |
 ## DELETE (all point to 35.212.86.161 — old host)
@@ -53,9 +55,8 @@
 | www.support.itpropartner.com | Moving to my.itpropartner.com |
 | mail.itpropartner.com | MXroute handles email |
 | ftp.itpropartner.com | Not used |
-| autodiscover.itpropartner.com | Exchange not used |
+|| autodiscover.itpropartner.com | Exchange not used |
-| autoconfig.itpropartner.com | Exchange not used |
+|| autoconfig.itpropartner.com | Exchange not used |
 | mainwp.itpropartner.com | Moving to ops |
 ## Service Routing Plan
@@ -0,0 +1,38 @@
 # Project Log — July 17, 2026
 ## WordPress Migration (wphost02 → app3)
 - **Status:** COMPLETE
 - 7 sites migrated from wphost02 (5.161.62.38) to app3 (152.53.241.111)
 - Sites: debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com, katiewattsdesign.com, vigilanttac.com, apextrackexperience.com, mainwp.itpropartner.com
 - Method: tar-pipe + mysqldump via SSH, CloudPanel site creation
 - DNS: Cloudflare for BoxPilot, IAmGMB, VigilantTAC, Katie, DRE; SiteGround for MainWP, Apex
 - Katie domain fix: katiewattsdesign.com (two t's in watts) on Cloudflare
 - Apex domain fix: apextrackexperience.com (not apexpredatorstackexperience)
 - DRE fix: debtrecoveryexperts.com (two r's in recovery), Nginx server_name typo
 - DRE SSL: Cloudflare Full mode, 525 resolved after Nginx fix
 ## Mattermost Deployment
 - **Status:** LIVE at noc.itpropartner.com (DNS pending)
 - **Server:** app1 (152.53.36.131), Docker, Team Edition
 - **Backend:** PostgreSQL 16 Alpine
 - **Caddy:** noc.itpropartner.com → 127.0.0.1:8065
 - **Storage:** /root/docker/mattermost
 - Admin: info@itpropartner.com
 ## Backup Restore System
 - **Status:** LIVE at my.itpropartner.com/backups
 - **Server:** app3 (152.53.241.111), Flask port 8090
 - **Features:** 2x daily snapshots (1 AM + 1 PM), 60-day retention, collapsible per-site view, restore modal, download links
 - **Storage:** /opt/backup-restore/snapshots/<domain>/
 - **Cron:** 0 1,13 * * *
 - **Caddy:** Core proxies my.itpropartner.com/backups → app3:8090
 ## Katie Watts DNS Fix
 - Domain: katiewattsdesign.com (two t's)
 - DNS: Cloudflare, updated A record to 152.53.241.111
 - Previously: was resolving to old IP, Cloudflare zone accessible
 ## Model Credits Reliability
 - DeepSeek balance: $59.29 (not exhausted)
 - Cron fixed to track deepseek-v4-pro instead of deepseek-chat
 - Local Ollama fallback: llama3.2:3b
@@ -0,0 +1,49 @@
 # App3 — Infrastructure Reference
 **Server:** netcup RS 4000 G12
 **IP:** 152.53.241.111
 **Provisioned:** July 16, 2026
 **Role:** Customer web hosting (WordPress sites)
 ## Software Stack
 | Component | Version |
 |---|---|
 | Debian | 13 (Trixie) |
 | Nginx | 2.x |
 | PHP | 8.3 (FPM) |
 | MySQL | Percona 8.4 |
 | CloudPanel CE | Latest |
 | Tailscale | Enabled |
 ## Hosted Sites
 | Domain | Type | Status |
 |---|---|---|
 | voipsimplicity.com | WordPress | ✅ Live |
 | support.voipsimplicity.com | DNS only | ✅ Created |
 ## Issues Encountered & Fixed (Jul 16)
 1. **"Hello World" default page** — WordPress files were copied to `/home/voipsimplicity/htdocs/www/` but CloudPanel's Nginx root was `/home/voipsimplicity/htdocs/voipsimplicity.com/`. Fixed by rsyncing files to correct directory.
 2. **panel.itpropartner.com 404** — Manual Nginx config (`panel.conf`) was overriding CloudPanel's generated config. Removed manual config. CloudPanel's reverse proxy template had `try_files $uri =404` instead of `try_files $uri @reverse_proxy` — fixed.
 3. **CloudPanel MFA blocking login** — MFA was enabled on the `admin` user by default. Disabled via `clpctl user:disable:mfa`.
 4. **Let's Encrypt failure (unproxied DNS)** — DNS for `panel.itpropartner.com` was on SiteGround nameservers, not Cloudflare. Added manually via SiteGround panel. CloudPanel SSL via HTTP-01 challenge required `.well-known/acme-challenge` directory at reverse proxy site root.
 5. **"Error establishing a database connection"** — MySQL auth plugin mismatch. Site DB user was using `caching_sha2_password` but WordPress expects `mysql_native_password`. Fixed by converting auth plugin.
 6. **itpropartner.com not on Cloudflare** — Nameservers still at SiteGround. All DNS changes for `*.itpropartner.com` must be done at SiteGround manually.
 ## Access
 - **CloudPanel:** https://panel.itpropartner.com — `info@itpropartner.com` / `CloudPanel2026`
 - **SSH:** root@152.53.241.111 (itpp-infra key)
 - **MySQL root:** `root` / `MOQMINFQIhklM0AF` on `127.0.0.1:3306`
 ## Pending
 - Migrate remaining WordPress sites from wphost02 to app3
 - Update all CloudPanel passwords (currently plaintext in this doc)