From a242be932403c7a3e51fcf43803c10135e50b7fd Mon Sep 17 00:00:00 2001 From: root Date: Fri, 17 Jul 2026 11:39:47 -0400 Subject: [PATCH] Jul 17: WP migration complete, Mattermost deployed, backup restore system, DNS updates --- CHANGELOG.md | 6 +++++ README.md | 16 +++++++++---- backup-plan.md | 8 +++++-- dns-records.md | 11 +++++---- project-log-2026-07-17.md | 38 ++++++++++++++++++++++++++++++ sites/app3.md | 49 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 117 insertions(+), 11 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 project-log-2026-07-17.md create mode 100644 sites/app3.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b51b5ee --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,6 @@ +# itpp-infrastructure — CHANGELOG + +## 2026-07-16 — Audit Remediation + +- Created CHANGELOG.md (missing per project documentation standard) +- Project directory: `/root/projects/itpp-infrastructure` diff --git a/README.md b/README.md index 0ead895..cccca9a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # IT Pro Partner — Infrastructure Documentation -> **Last Updated:** July 16, 2026 +> **Last Updated:** July 17, 2026 > **Maintainer:** Sho'Nuff --- @@ -42,8 +42,10 @@ - Open WebUI (Docker, port 3000) — ai.itpropartner.com - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com + - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com - Caddy (systemd, 80/443) - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903) + - Super Search MCP (systemd, port 8899) ### App2 Server - **Hostname:** app2 @@ -65,12 +67,16 @@ - **Provider:** netcup RS 4000 G12 - **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe - **OS:** Debian 13 -- **Role:** Web Hosting +- **Role:** Web Hosting + Backup Restore - **Key Services:** - CloudPanel CE — panel.itpropartner.com - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3 - - voipsimplicity.com (WordPress, migrated Jul 16) - - my.voipsimplicity.com (static HTML SPA, VoIP customer portal) + - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups + - WordPress sites (7 migrated from wphost02, all live): + - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com + - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com + - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com + - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots ### Core-BU (Warm Standby) - **Hostname:** core-bu @@ -249,3 +255,5 @@ All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-rene - **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously. - **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306. - **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL. +- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0. +- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise. diff --git a/backup-plan.md b/backup-plan.md index f0814e0..176c88e 100644 --- a/backup-plan.md +++ b/backup-plan.md @@ -19,6 +19,7 @@ | 2:30 AM | app2 | /root/backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi | | 3:00 AM | Core | root-essentials-backup.sh | /root essentials | | 3:00 AM | app3 | /root/backup.sh | CloudPanel, MySQL, WordPress, Nginx configs | +| 1:00 AM + 1:00 PM | app3 | /opt/backup-restore/snapshot.sh | Per-site WordPress snapshots (files + DB), 60-day retention | | 3:00 AM | Core | docker-volume-sync.sh | Docker volumes | | 4:00 AM | Core | system-config-sync.sh | System configs | | 6:00 AM | Core | home-router-backup.sh | MikroTik CCR config | @@ -51,8 +52,9 @@ | LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr | | Open WebUI | /app/backend/data → S3 (daily) | 24 hr | | n8n | Postgres pg_dump → S3 (daily) | 24 hr | -| MCP servers | Config files → S3 (daily) | 24 hr | -| Ollama | Model list → S3 (daily) | 24 hr | +| Ollama | Models preserved in Docker volume | 24 hr | +| MCP servers | Configs in Git + Docker volumes | 24 hr | +| Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr | ### app2 (152.53.39.202) @@ -72,6 +74,8 @@ | MySQL | All databases mysqldump → S3 (daily) | 24 hr | | WordPress | Files + wp-config → S3 (daily) | 24 hr | | Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr | +| **WordPress Snapshots** | **Per-site tarball + MySQL dump (2x daily, 60-day retention)** | **12 hr** | +| **Backup Restore UI** | **Flask app port 8090 at my.itpropartner.com/backups** | **Instant** | | voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr | | my.voipsimplicity.com | Covered by Git (static HTML) | N/A | diff --git a/dns-records.md b/dns-records.md index d854079..e2ba671 100644 --- a/dns-records.md +++ b/dns-records.md @@ -1,13 +1,12 @@ # DNS Records — itpropartner.com **Managed by:** SiteGround Site Tools (no API access) -**Updated:** July 16, 2026 +**Updated:** July 17, 2026 **⚠️ Germaine must apply changes manually at SiteGround** ## CREATE (pending) | Subdomain | Record | IP | Service | |---|---|---|---| -| my.itpropartner.com | A | 152.53.192.33 | Customer portal hub | | vault.itpropartner.com | A | 152.53.36.131 | Vaultwarden (moving from Core) | | status.itpropartner.com | A | 152.53.192.33 | Public status page | @@ -23,18 +22,21 @@ |---|---|---|---| | ops.itpropartner.com | 152.53.192.33 | Core | Operations dashboard | | core.itpropartner.com | 152.53.192.33 | Core | Landing page + MikroTik | +| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub | | sign.itpropartner.com | 152.53.192.33 | Core | Document signing | | uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring | | app.itpropartner.com | 152.53.192.33 | Core | App entry | | admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM | | ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI | | n8n.itpropartner.com | 152.53.36.131 | app1 | Workflow automation | +| **noc.itpropartner.com** | **152.53.36.131** | **app1** | **Mattermost NOC chat** | | app1.itpropartner.com | 152.53.36.131 | app1 | App server 1 | | git.itpropartner.com | 152.53.39.202 | app2 | Gitea | | hudu.itpropartner.com | 152.53.39.202 | app2 | Documentation | | unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller | | www.unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi (www) | | panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel | +| mainwp.itpropartner.com | 152.53.241.111 | app3 | MainWP management | ## DELETE (all point to 35.212.86.161 — old host) @@ -53,9 +55,8 @@ | www.support.itpropartner.com | Moving to my.itpropartner.com | | mail.itpropartner.com | MXroute handles email | | ftp.itpropartner.com | Not used | -| autodiscover.itpropartner.com | Exchange not used | -| autoconfig.itpropartner.com | Exchange not used | -| mainwp.itpropartner.com | Moving to ops | +|| autodiscover.itpropartner.com | Exchange not used | +|| autoconfig.itpropartner.com | Exchange not used | ## Service Routing Plan diff --git a/project-log-2026-07-17.md b/project-log-2026-07-17.md new file mode 100644 index 0000000..146f613 --- /dev/null +++ b/project-log-2026-07-17.md @@ -0,0 +1,38 @@ +# Project Log — July 17, 2026 + +## WordPress Migration (wphost02 → app3) +- **Status:** COMPLETE +- 7 sites migrated from wphost02 (5.161.62.38) to app3 (152.53.241.111) +- Sites: debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com, katiewattsdesign.com, vigilanttac.com, apextrackexperience.com, mainwp.itpropartner.com +- Method: tar-pipe + mysqldump via SSH, CloudPanel site creation +- DNS: Cloudflare for BoxPilot, IAmGMB, VigilantTAC, Katie, DRE; SiteGround for MainWP, Apex +- Katie domain fix: katiewattsdesign.com (two t's in watts) on Cloudflare +- Apex domain fix: apextrackexperience.com (not apexpredatorstackexperience) +- DRE fix: debtrecoveryexperts.com (two r's in recovery), Nginx server_name typo +- DRE SSL: Cloudflare Full mode, 525 resolved after Nginx fix + +## Mattermost Deployment +- **Status:** LIVE at noc.itpropartner.com (DNS pending) +- **Server:** app1 (152.53.36.131), Docker, Team Edition +- **Backend:** PostgreSQL 16 Alpine +- **Caddy:** noc.itpropartner.com → 127.0.0.1:8065 +- **Storage:** /root/docker/mattermost +- Admin: info@itpropartner.com + +## Backup Restore System +- **Status:** LIVE at my.itpropartner.com/backups +- **Server:** app3 (152.53.241.111), Flask port 8090 +- **Features:** 2x daily snapshots (1 AM + 1 PM), 60-day retention, collapsible per-site view, restore modal, download links +- **Storage:** /opt/backup-restore/snapshots// +- **Cron:** 0 1,13 * * * +- **Caddy:** Core proxies my.itpropartner.com/backups → app3:8090 + +## Katie Watts DNS Fix +- Domain: katiewattsdesign.com (two t's) +- DNS: Cloudflare, updated A record to 152.53.241.111 +- Previously: was resolving to old IP, Cloudflare zone accessible + +## Model Credits Reliability +- DeepSeek balance: $59.29 (not exhausted) +- Cron fixed to track deepseek-v4-pro instead of deepseek-chat +- Local Ollama fallback: llama3.2:3b diff --git a/sites/app3.md b/sites/app3.md new file mode 100644 index 0000000..dc7b810 --- /dev/null +++ b/sites/app3.md @@ -0,0 +1,49 @@ +# App3 — Infrastructure Reference + +**Server:** netcup RS 4000 G12 +**IP:** 152.53.241.111 +**Provisioned:** July 16, 2026 +**Role:** Customer web hosting (WordPress sites) + +## Software Stack + +| Component | Version | +|---|---| +| Debian | 13 (Trixie) | +| Nginx | 2.x | +| PHP | 8.3 (FPM) | +| MySQL | Percona 8.4 | +| CloudPanel CE | Latest | +| Tailscale | Enabled | + +## Hosted Sites + +| Domain | Type | Status | +|---|---|---| +| voipsimplicity.com | WordPress | ✅ Live | +| support.voipsimplicity.com | DNS only | ✅ Created | + +## Issues Encountered & Fixed (Jul 16) + +1. **"Hello World" default page** — WordPress files were copied to `/home/voipsimplicity/htdocs/www/` but CloudPanel's Nginx root was `/home/voipsimplicity/htdocs/voipsimplicity.com/`. Fixed by rsyncing files to correct directory. + +2. **panel.itpropartner.com 404** — Manual Nginx config (`panel.conf`) was overriding CloudPanel's generated config. Removed manual config. CloudPanel's reverse proxy template had `try_files $uri =404` instead of `try_files $uri @reverse_proxy` — fixed. + +3. **CloudPanel MFA blocking login** — MFA was enabled on the `admin` user by default. Disabled via `clpctl user:disable:mfa`. + +4. **Let's Encrypt failure (unproxied DNS)** — DNS for `panel.itpropartner.com` was on SiteGround nameservers, not Cloudflare. Added manually via SiteGround panel. CloudPanel SSL via HTTP-01 challenge required `.well-known/acme-challenge` directory at reverse proxy site root. + +5. **"Error establishing a database connection"** — MySQL auth plugin mismatch. Site DB user was using `caching_sha2_password` but WordPress expects `mysql_native_password`. Fixed by converting auth plugin. + +6. **itpropartner.com not on Cloudflare** — Nameservers still at SiteGround. All DNS changes for `*.itpropartner.com` must be done at SiteGround manually. + +## Access + +- **CloudPanel:** https://panel.itpropartner.com — `info@itpropartner.com` / `CloudPanel2026` +- **SSH:** root@152.53.241.111 (itpp-infra key) +- **MySQL root:** `root` / `MOQMINFQIhklM0AF` on `127.0.0.1:3306` + +## Pending + +- Migrate remaining WordPress sites from wphost02 to app3 +- Update all CloudPanel passwords (currently plaintext in this doc)