Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,569 @@
+---
+name: ubiquiti-controller-management
+description: Umbrella for Ubiquiti UISP (UNMS) and UniFi Network controller deployment, migration, and backup operations.
+---
+
+This skill covers the installation, migration, and maintenance of Ubiquiti infrastructure controllers (UISP/UNMS and UniFi).
+
+## UniFi Admin Password Reset via MongoDB (Verified v10.0.162, Jul 2026)
+
+When locked out of the UniFi controller, create an admin account directly in MongoDB. The `jacobalberty/unifi` image runs MongoDB on port **27117** (not 27017), bound to 127.0.0.1. The shell binary varies by image version: **`mongo`** on older builds, **`mongosh`** on newer ones. Always check which is available:
+
+```bash
+docker exec unifi-controller which mongosh 2>/dev/null && echo 'mongosh' || echo 'mongo'
+```
+
+**Pitfall:** Some servers (e.g. Hetzner bare-metal with UniFi installed via dpkg) have `mongosh` but no `mongo` binary at all. Running `mongo` returns `No such file or directory` (exit 127) even though MongoDB is running on 27117. Use the correct shell for the server.
+
+### Steps
+
+1. **Check existing admins:**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   print("admins: " + db.admin.count());
+   print("sites: " + db.site.count());
+   '
+   ```
+
+2. **Generate a SHA-512 crypt password hash:**
+   ```bash
+   openssl passwd -6 'TempPassword123!'
+   # Output: $6$salt$hash...
+   ```
+
+3. **Insert the admin:**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   var now = Math.floor(new Date().getTime() / 1000);
+   db.admin.insertOne({
+       name: "Admin Name",
+       email: "admin@example.com",
+       x_shadow: "$6$...",
+       time_created: now,
+       last_site_name: "default",
+       super_site_permissions: ["super"],
+       super_site_role: "super",
+       is_super: true
+   });
+   '
+   ```
+
+4. **Create privilege records (REQUIRED — without these, the UI is blank):**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   var admin = db.admin.findOne({email: "admin@example.com"});
+   var sites = db.site.find({}).toArray();
+   sites.forEach(function(site) {
+       db.privilege.insertOne({
+           admin_id: admin._id,
+           site_id: site._id,
+           role: "SUPER_ADMIN",
+           permissions: ["ALL"],
+           is_super: true
+       });
+   });
+   print("Privileges created: " + db.privilege.count());
+   '
+   ```
+
+### Critical Pitfalls
+
+- **Admin `_id` and site `_id` MUST be ObjectIds, not strings** in the privilege records. Using `.str` will cause the admin to see a blank page after login.
+- **`super_site_permissions` must be `["super"]`** — an empty array means no site access.
+- **`is_super: true`** must be set on both the admin document and privilege records.
+- If the admin was created before privileges, the existing session cookie caches the old (empty) permissions. Log out and log back in.
+
+### Verification
+
+```bash
+curl -sk -c /tmp/cookie -X POST https://localhost:8443/api/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"admin@example.com","password":"TempPassword123!"}'
+# Expect: {"meta":{"rc":"ok"}}
+
+curl -sk -b /tmp/cookie https://localhost:8443/api/self
+# Expect: admin details with is_super:true
+```
+
+### How UniFi Stores Users
+
+| Collection | Purpose |
+|---|---|
+| `ace.admin` | Admin accounts (name, email, x_shadow hash) |
+| `ace.privilege` | Links admins to sites with roles |
+| `ace.site` | Sites ("super" and "default") |
+| `ace.account` | Local device/guest accounts (not admins) |
+
+See `references/unifi-mongodb-admin-schema.md` for full field reference.
+
+## UniFi Backup File Format
+
+UniFi `.unf` backup files are **encrypted BSON** — not ZIP files. Python's `zipfile` and `bsondump` both fail. The backups can **only** be decrypted and imported by the UniFi `ace.jar restore` command. There is no third-party tool to extract `.unf` contents.
+
+### Restoring via one-off container (ace.jar restore)
+
+The `ace.jar restore` command initializes a full Spring application context. It CANNOT run while the UniFi service is active — the two contexts collide. Stop the main container, then run restore in an isolated one-off that shares volumes:
+
+```bash
+# 1. Copy .unf backup into the anonymous volume
+docker cp backup.unf unifi-controller:/unifi/data/backup/autobackup/
+docker stop unifi-controller
+
+# 2. Run restore in one-off with --volumes-from
+docker run --rm --volumes-from unifi-controller \
+  --entrypoint /bin/bash jacobalberty/unifi:latest \
+  -c "
+    mongod --dbpath /unifi/data/db --port 27117 --bind_ip 127.0.0.1 --fork --logpath /tmp/mongod.log
+    sleep 3
+    java -jar /usr/lib/unifi/lib/ace.jar restore /unifi/data/backup/autobackup/backup.unf
+    mongosh --port 27117 admin --eval 'db.shutdownServer()'
+  "
+
+# 3. Start main container
+docker start unifi-controller
+```
+
+**Pitfall — quoting hell:** When running this through SSH with nested quotes, write the restore commands to a script file and mount it into the one-off container (`-v /tmp/restore.sh:/tmp/restore.sh:ro`). Shell escaping across SSH → Docker → bash cascades is fragile.
+
+**Pitfall — Spring context collision:** Without `--volumes-from`, the one-off sees no `/unifi/data/db` (the path inside the container is `/unifi/data/db`, not `/usr/lib/unifi/data/db` — the latter is a symlink created by the entrypoint). Using only `-v /opt/unifi:/config` misses the live MongoDB data entirely.
+
+See `references/unifi-backup-restore-pitfalls.md` for additional restore dead-ends encountered with v10.x.
+
+## `jacobalberty/unifi` Dual-Storage Architecture
+
+This Docker image uses TWO separate storage locations:
+
+| Mount | Host Path | Container Path | Purpose |
+|---|---|---|---|
+| Bind mount | `/opt/unifi` | `/config` | system.properties, keystore, firmware.json, backup/ |
+| Anonymous volume | Docker-managed | `/unifi` | MongoDB data (db/), backup/ (live), sites/ |
+
+**These can drift apart.** The `/unifi` anonymous volume holds the current MongoDB, while `/config` holds config files placed during deployment. After a container restart, `/unifi/data` gets fresh MongoDB while `/config` keeps the old files. The container uses `/unifi/data/` as the live data directory (symlinked from `/usr/lib/unifi/data` → `/unifi/data`).
+
+**When running a one-off restore container**, you need `--volumes-from` to access the anonymous volume, not just `-v /opt/unifi:/config`. The data path inside the container for mongod is `/unifi/data/db`, not `/usr/lib/unifi/data/db`.
+
+## UniFi Site Manager vs Local Network API
+
+The UniFi Site Manager API (`https://api.ui.com/v1` or `/ea`) is useful for high-level inventory only: sites, hosts, device summaries, WAN issue summaries. It does **not** expose enough detail for switch-port troubleshooting or PoE actions: port counters, client history, events, PoE cycle, CRC/drop data.
+
+For port-level troubleshooting or automation on UDM/UDM-Pro sites, create a **local Network API key** inside each console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+Then use the console-local Network API for clients/devices/ports/actions. The port action endpoint shape documented by UniFi is:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+Use read-only checks first. Do not power-cycle PoE or ports without explicit user approval. See `references/unifi-local-network-api-port-troubleshooting.md` for the Liberty/GLC identifiers and Yealink W70 workflow pattern.
+
+## UniFi Network Controller Deployment
+
+### PREFERRED: `jacobalberty/unifi:latest` (verified Jul 12, 2026 on app2)
+
+**This is the only image that works reliably.** Bundles MongoDB internally — single `docker run`, no external DB, no auth setup. First boot ~30s to HTTP 200 on 8443. Container healthcheck works out of box.
+
+### Simple Deployment (app2 reference — verified working)
+
+```bash
+docker run -d --restart unless-stopped \
+  --name unifi-controller \
+  -p 8443:8443 \
+  -p 8080:8080 \
+  -p 8880:8880 \
+  -p 8843:8843 \
+  -p 3478:3478/udp \
+  -p 10001:10001/udp \
+  -v /opt/unifi:/config \
+  -e TZ=America/New_York \
+  -e RUNAS_UID0=true \
+  jacobalberty/unifi:latest
+```
+
+Wait 30-60 seconds for first boot, then access at `https://<host>:8443`. Import a `.unf` backup file through the setup wizard to restore from an old controller.
+
+### ALTERNATIVE (complex, NOT recommended): `linuxserver/unifi-network-application`
+
+**Requires external MongoDB 7.0 with auth.** This image does NOT bundle MongoDB internally.
+Bundles MongoDB internally — single `docker run`, no external DB, no auth setup.
+- First boot ~30s to HTTP 200 on 8443
+- Container healthcheck works out of box
+- **Verified working on app2 (152.53.39.202) Jul 12 2026**
+
+**Option B: `linuxserver/unifi-network-application` (external MongoDB REQUIRED)**
+This image does NOT bundle MongoDB. Must provide external MongoDB 7.0 with auth.
+
+**CRITICAL PITFALL with Option B:** The `linuxserver/unifi-network-application` image has a confirmed bug where `MONGO_USER` and `MONGO_PASS` environment variables are NOT honored by the Java application. Even with correctly configured MongoDB auth and all env vars set (`MONGO_USER`, `MONGO_PASS`, `MONGO_HOST`, `MONGO_PORT`, `MONGO_DBNAME`, even `MONGO_URI`), the controller throws:
+```
+java.lang.IllegalArgumentException: No username is provided in the connection string
+```
+This was tested with: MongoDB 7.0 with auth, MongoDB 7.0 without auth, explicit `MONGO_URI=mongodb://user:pass@host:27017/db`, and the individual `MONGO_USER`/`MONGO_PASS` env vars. None worked.
+
+**Fix:** Switch to Option A (`jacobalberty/unifi`). It bundles MongoDB with the correct auth setup and works on first boot. The linuxserver image's Java connection string construction ignores the user/pass env vars, making it unusable for any deployment that requires authentication.
+
+### MongoDB Requirement
+
+Starting with version 8.1, UniFi Network Application supports MongoDB 3.6 through 7.0. Version 9.0+ also supports MongoDB 8.0. Pin your MongoDB version — do NOT use `latest`, as MongoDB does not support automatic upgrades between major versions. Use `mongo:7.0` for broad compatibility.
+
+### Password Encoding (Critical Pitfall)
+
+**Special characters in the MongoDB password MUST be URL-encoded** in the `MONGO_PASS` env var passed to the UniFi container. Characters that break the MongoDB connection string if not encoded: `@` → `%40`, `:` → `%3A`, `%` → `%25`, `!` → `%21`. The easiest approach is to use **alphanumeric-only passwords** (no special chars) for both the MongoDB init script and the UniFi container to avoid encoding issues entirely.
+
+When you do need special characters, encode them in `MONGO_PASS` only — the MongoDB init script uses the raw password to create the user, but the UniFi container's Java process builds a MongoDB connection string from the env var and needs it encoded.
+
+### Port Mappings
+
+| Port | Purpose | Required |
+|------|---------|----------|
+| 8443 | Web admin UI (HTTPS) | Yes |
+| 8080 | Device communication | Yes |
+| 3478/udp | STUN | Yes |
+| 10001/udp | AP discovery | Yes |
+| 8880 | Guest portal HTTP redirect | Optional |
+| 8843 | Guest portal HTTPS redirect | Optional |
+| 1900/udp | L2 controller discoverability | Optional |
+| 6789 | Mobile throughput test | Optional |
+| 5514/udp | Remote syslog | Optional |
+
+**Do NOT use ports 443 or 80** — the UniFi controller does not need them, and they commonly conflict with Caddy or other web services. The controller uses self-signed TLS on 8443; device inform uses HTTP on 8080.
+
+### Access and First-Run Wizard
+
+The web UI is at `https://<host>:8443`. On first run, use the setup wizard to either configure fresh or restore a backup. UniFi takes 2-3 minutes to fully initialize on first launch (MongoDB init + Java application startup). Port 8443 will be listening before the app is ready; curl will get TLS errors until initialization completes.
+
+Monitor progress with: `docker logs -f unifi-controller`
+
+### Post-Deployment Verification
+
+```bash
+docker ps --filter name=unifi --format 'table {{.Names}}\\t{{.Status}}'
+curl -sk https://localhost:8443/ | head -5
+```
+
+**Pitfall — API returns HTML during initialization:** The `/api/login` and all other API endpoints return the "Network application is starting up..." HTML status page (not JSON) while UniFi initializes. Curl calls to any API endpoint will get full HTML pages during this window. The `/status` endpoint is the authoritative check — wait for `\"up\":true` before calling any API. On first boot this is 30-60 seconds; on container restart, 5-15 seconds. Loop until ready:
+
+## Caddy Reverse Proxy for UniFi (Self-Signed Cert)
+
+When proxying UniFi behind Caddy with a public domain (e.g., `unifi.itpropartner.com` → `localhost:8443`), UniFi's self-signed TLS cert requires `tls_insecure_skip_verify` and HTTP transport mode:
+
+```caddy
+unifi.itpropartner.com {
+    reverse_proxy localhost:8443 {
+        transport http {
+            tls_insecure_skip_verify
+        }
+    }
+}
+```
+
+**Without `tls_insecure_skip_verify`**, Caddy returns `TLS alert, internal error` when trying to connect to UniFi's self-signed cert. The **browser sees a blank page** — no error, no content, just nothing.
+
+**With the correct config**, Caddy terminates public TLS (Let's Encrypt), then proxies HTTP to UniFi on 8443 with cert verification disabled. The browser gets valid HTTPS.
+
+**Pitfall:** `curl -H 'Host: unifi.itpropartner.com' https://<server-ip>/` tests Caddy's routing but NOT SNI. Use the actual domain to test: `curl -skL https://unifi.itpropartner.com/`. When the proxy works, you'll get `HTTP/2 302 → /manage` followed by the UniFi SPA HTML.
+
+### Access and First-Run Wizard
+
+The web UI is at `https://<host>:8443`. On first run, use the setup wizard to either configure fresh or restore a backup. UniFi takes 2-3 minutes to fully initialize on first launch (MongoDB init + Java application startup). Port 8443 will be listening before the app is ready; curl will get TLS errors until initialization completes.
+
+Monitor progress with: `docker logs -f unifi-controller`
+
+## UNMS/UISP Backup to S3 — Dual-Source Archive (updated Jul 12, 2026)
+
+Syncs backup from BOTH the old server (historical archive) AND the live server (current operations) to S3.
+
+**Old server (historical):** `5.161.225.131` → `s3://hermes-vps-backups/unms-backups/old-unms/`
+**Live server (current):** `152.53.39.202` (app2) → `s3://hermes-vps-backups/unms-backups/live/`
+
+**Backup location on both servers:** `/home/unms/data/unms-backups/backups/`
+**Backup files:** Named `YYYYMMDDHHMMSS-auto-<version>.unms`, ~140MB each
+**Backup frequency:** Nightly auto-backup around 4:00 AM
+**UCRM backup also at:** `/home/unms/data/ucrm/ucrm/data/backup/`
+
+**S3 sync script pattern (saved at /root/.hermes/scripts/unms-backup-sync.sh):**
+
+Do **not** run `rsync` directly to an `s3://...` URL. `rsync` does not speak S3; that pattern silently fails or copies nothing useful. Use a two-step pipeline:
+
+1. Install `rsync` on both the agent host and the remote UISP/UNMS host.
+2. `rsync` remote backup directories to local staging under root disk, not `/tmp`.
+3. Upload staging to Wasabi with `aws s3 sync --endpoint-url https://s3.us-east-1.wasabisys.com`.
+4. Verify S3 has at least one object under the live backup prefix before returning success.
+
+```bash
+STAGE="/root/.hermes/.backups/unms-sync"
+S3_BUCKET="s3://hermes-vps-backups/unms-backups"
+ENDPOINT="https://s3.us-east-1.wasabisys.com"
+SSH_KEY="/root/.ssh/itpp-infra"
+
+rsync -az --delete -e "ssh -i ${SSH_KEY} -o BatchMode=yes" \
+  root@152.53.39.202:/home/unms/data/unms-backups/backups/ \
+  "$STAGE/live/backups/"
+
+aws s3 sync "$STAGE/" "$S3_BUCKET/" --endpoint-url "$ENDPOINT" --delete
+aws s3 ls "$S3_BUCKET/live/backups/" --endpoint-url "$ENDPOINT" | grep .
+```
+
+**Cron schedule:** Daily at 6 AM ET, no_agent mode, script-only.
+Created via: `cronjob(action='create', name='unms-backup-sync', schedule='0 6 * * *', script='unms-backup-sync.sh', no_agent=True)`
+
+**Verification:** Check S3 bucket has files after first run:
+```bash
+aws s3 ls --recursive s3://hermes-vps-backups/unms-backups/ \
+  --endpoint-url https://s3.us-east-1.wasabisys.com | tail -5
+```
+
+**This captures 3 data streams:**
+1. Full UISP .unms auto-backups (auto every ~24h, manual download also works)
+2. UCRM database backups (found at `/home/unms/data/ucrm/ucrm/data/backup/database/`)
+3. Config snapshots (UUID-labeled directories at `/home/unms/data/config-backups/`)
+
+## UNMS Nginx Port Architecture (Critical for Reverse Proxies)
+
+The UNMS nginx container (`ubnt/unms-nginx`) listens on multiple ports, each with a distinct purpose. **Misrouting web traffic to the wrong port causes cryptic errors.**
+
+| Port | Protocol | Purpose | Proxy-Ready |
+|------|----------|---------|-------------|
+| 80 (internal) | HTTP | ACME challenges only; redirects to 443 | ❌ Redirect loop |
+| 443 (internal) | HTTPS | **Main web UI** — `/nms/login`, full UNMS/UCRM | ✅ **Proxy here** |
+| 81 | HTTP | UCRM suspend page | ❌ 503 for `/nms/*` |
+| 8089 | HTTPS (SSL) | **WebSocket tunnel** (`REMOTE_UI_WS_PORT_PUBLIC`) — always sends `Upgrade` header to `unms-api:8085` | ❌ **NEVER proxy web traffic here** |
+
+### The Port 8089 Trap
+
+Port 8089 is often the only HTTPS port exposed from the nginx container (docker-compose default: only `81:81` and `8089:8089`). It's tempting to point a reverse proxy at it. **Don't.** This port:
+
+- Is `listen 8089 ssl` — expects TLS from the client
+- Has **one location block** that ALWAYS sets `proxy_set_header Upgrade` and `Connection: upgrade`
+- Proxies to `unms-api:8085` (WebSocket endpoint, not the web UI)
+- Returns `"Upgrade Required"` (426) for regular HTTP requests
+- Returns `"400 The plain HTTP request was sent to HTTPS port"` if you send plain HTTP to it
+
+**Symptoms of misrouting to 8089:**
+- `curl https://unms.forefrontwireless.com/nms/login` → HTTP 426 "Upgrade Required"
+- `curl http://localhost:8089` → HTTP 400 "The plain HTTP request was sent to HTTPS port" (from OpenResty)
+
+### Exposing the Real Web UI (Port 443)
+
+The UNMS web UI lives on the container's **internal port 443**, which is NOT exposed by default. Add it to the nginx service in `/home/unms/app/docker-compose.yml`:
+
+```yaml
+nginx:
+  ports:
+    - 81:81
+    - 8089:8089
+    - 8444:443    # ← ADD THIS (use port that doesn't conflict)
+```
+
+**Port selection:** On app2, 8443 is taken by UniFi. Use 8444 or another free port. Check with `ss -tlnp | grep 844` before choosing.
+
+After editing the compose file, recreate the container:
+
+```bash
+docker stop unms-nginx && docker rm unms-nginx
+docker compose -p unms -f /home/unms/app/docker-compose.yml up -d nginx
+```
+
+**Pitfall — compose project name mismatch:** The UNMS installer creates containers with project name `unms` (networks: `unms_public`, `unms_internal`), but the compose file lives in `/home/unms/app/`. Running `docker compose` without `-p unms` defaults to project `app`, which tries to create `app_public`/`app_internal` networks and fails with `invalid pool request: Pool overlaps with other one on this address space`. **Always use `docker compose -p unms`** when managing UNMS services.
+
+### Caddy Reverse Proxy for UNMS (Self-Signed Cert)
+
+UNMS uses a self-signed or auto-generated cert on its internal nginx. Same pattern as UniFi — Caddy terminates public TLS, then proxies via HTTPS with `tls_insecure_skip_verify`:
+
+```caddy
+unms.forefrontwireless.com {
+    reverse_proxy https://localhost:8444 {
+        transport http {
+            tls_insecure_skip_verify
+        }
+    }
+}
+```
+
+**Verification:**
+```bash
+# Backend reachable:
+curl -sk -o /dev/null -w '%{http_code}' https://localhost:8444/nms/login
+# → 200
+
+# Public URL works:
+curl -sk -o /dev/null -w '%{http_code}' https://unms.forefrontwireless.com/nms/login
+# → 200
+```
+
+## netcup Firewall: Ports 80/443 Blocked by UFW
+
+**netcup RS-series servers ship with UFW enabled and only port 22 open by default.** This blocks all inbound HTTP/HTTPS traffic, making UNMS/UISP unreachable from the internet despite Docker containers being configured correctly.
+
+### Symptom
+- UNMS Nginx Docker container is running and listening on ports 80/443 internally
+- `curl http://localhost:80` works from the server itself
+- `curl http://<public-ip>:80` times out from any external host
+- Certbot HTTP challenge fails with "Timeout during connect (likely firewall problem)"
+- Browser shows ERR_CONNECTION_TIMED_OUT or SSL errors
+
+### Diagnosis
+```bash
+ufw status
+# Output shows only: 22/tcp ALLOW Anywhere
+ufw status verbose
+# Shows default policy: DROP on incoming
+```
+
+### Fix
+```bash
+ufw allow 80/tcp comment 'UNMS HTTP'
+ufw allow 443/tcp comment 'UNMS HTTPS'
+ufw reload
+# Verify:
+ufw status | head -10
+```
+
+### After Opening Ports
+1. UNMS Nginx becomes accessible from the internet
+2. Certbot's HTTP challenge succeeds (no port 80 conflict)
+3. Let's Encrypt certificates can be obtained for `unms.forefrontwireless.com` and `unifi.itpropartner.com`
+4. The UNMS Nginx must be stopped briefly for certbot standalone mode:
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com \
+  --preferred-challenges http
+docker start unms-nginx
+```
+
+### Certificate Injection into UNMS
+Certbot stores certs at `/etc/letsencrypt/live/<domain>/`. Inject them into UNMS:
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+Certbot auto-renewal via systemd timer is set up by default on Debian 13.
+
+### All netcup servers May Have This Issue
+This may apply to ALL netcup RS-series VPS, not just app2. When deploying any web service on a new netcup box, check UFW immediately:
+```bash
+ufw status | grep -q '80.*ALLOW' || echo "Port 80 blocked!"
+```
+
+### UNMS/UISP Backup Restore — WebSocket Container Recovery (Jul 12, 2026)
+
+After restoring a `.unms` backup to a new server, the UNMS `device-ws` containers (which handle WebSocket connections from devices) may be in `Created` state but NOT `Running`. This causes devices to fail registration with:
+
+```
+connection error (host:443): HS: ws upgrade response not 101
+```
+
+The device connects to UNMS on port 443, but since the WebSocket handler containers aren't running, the Nginx returns a regular HTTP response instead of a WebSocket upgrade (101). The device error log shows 10 connection attempts, all failing with "ws upgrade response not 101".
+
+**Diagnosis:**
+```bash
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# Output: unms-device-ws-1 Created  (NOT "Up" — not running)
+```
+
+**Fix — start all device-ws containers:**
+```bash
+for i in 1 2 3 4 5 6 7 8 9 10 11; do
+  docker start unms-device-ws-$i
+done
+# Verify all healthy:
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# All should show "Up X seconds (healthy)"
+```
+
+**After starting, re-register the device with the connection string.**
+
+### UNMS/UISP Backup Restore — Vault Key Missing
+
+After restoring a backup to a new server, the Credentials Vault key from the original installation is required. This key was set once during initial UISP setup and is NOT stored in any config file, .env, or database. It is recoverable only if the original operator wrote it down.
+
+**Error:** "Vault Key Is Missing" when accessing device credentials in the UISP UI.
+
+**Options:**
+1. **Destroy vault → create new** — UISP will auto-generate new passwords for online devices. Offline devices need manual updates. Use UISP web UI: Settings → Credentials → Vault → Destroy Vault.
+2. **Use old server as reference** — Keep the old UISP server running simultaneously while migrating, so device configs (IP, radios, names) can be looked up during manual re-registration.
+
+**Destroy path is the intended recovery mechanism** per UISP documentation: https://help.ui.com/hc/en-us/articles/360019506834
+
+### Certificate Injection After Backup Restore
+
+After restoring a backup to a new server, the UNMS Nginx will use self-signed or localhost certificates from the restored data. These must be replaced with valid Let's Encrypt certs.
+
+**Prerequisites:** UFW must have ports 80 and 443 open (see netcup UFW section above).
+
+**Step 1 — Get Let's Encrypt certs:**
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com \
+  --preferred-challenges http
+docker start unms-nginx
+```
+
+**Step 2 — Inject certs into UNMS:**
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+**Step 3 — Verify:**
+```bash
+curl -sI --connect-timeout 5 https://unms.forefrontwireless.com/nms/login | head -3
+# Expected: HTTP/2 200, x-unms-login-screen: 1
+```
+
+Certbot auto-renewal runs daily via systemd timer on Debian 13.
+
+### UNMS/UISP Backup Restore — Full Recovery Sequence (Jul 12, 2026)
+
+When migrating UNMS to a new server, the complete sequence is:
+
+1. **Prerequisite:** UFW ports 80/443 open on the new server
+2. Obtain Let's Encrypt cert (standalone mode, brief nginx stop)
+3. Inject cert into /home/unms/data/cert/{live.crt,live.key}
+4. Start all device-ws containers (they may be Created but not Running)
+5. Accept the vault key is lost — destroy and recreate via web UI
+6. Import your .unms backup through the UISP admin UI
+7. Re-register devices manually with the new connection string
+8. Set up daily S3 backup sync (cron, 6 AM ET)
+
+### Database Backup & Restore
+* **Do NOT use `pg_dump`**: The live `unms-postgres` container uses TimescaleDB with complex hypertable/chunk circular foreign keys. A standard `pg_dump` will fail to restore.
+* **Use Native Backups**: Always use the native `.unms` auto-backups located at `/home/unms/data/unms-backups/backups/`.
+* **Restore path**: Place the backup file in `/home/unms/data/unms-backups/restore/` and run `/home/unms/app/unms-cli restore-backup --file <path>`.
+
+### Docker Version Pinning Pitfalls
+The official installer (`https://uisp.ui.com/v1/install`) has strict, hardcoded Docker version requirements (e.g., demanding 27.5.1 and failing on 29.6.1). This breaks automated unattended installs and interactive restores.
+
+**To deploy on a server with a newer Docker version:**
+1. Download the installer: `curl -fsSL https://uisp.ui.com/v1/install > /tmp/uisp_inst.sh && chmod +x /tmp/uisp_inst.sh`
+2. Run it once so it downloads and unpacks `/tmp/unms-install/install-full.sh` (it will fail the version check).
+3. Patch the unpacked installer to bypass checks:
+   ```bash
+   sed -i 's/SKIP_DOCKER_INSTALL="false"/SKIP_DOCKER_INSTALL="true"/g' /tmp/unms-install/install-full.sh
+   sed -i 's/if ! is_desired_docker_version_installed; then/if false; then/g' /tmp/unms-install/install-full.sh
+   cd /tmp/unms-install && ./install-full.sh --unattended
+   ```
+
+**To restore a backup bypassing the `unms-cli` version prompt:**
+`unms-cli restore-backup` will also interactively block on the Docker version mismatch. Bypass it by patching the CLI script before restoring:
+```bash
+sed -i "s/check_docker_version/true/g" /home/unms/app/unms-cli
+sed -i "s/confirm \\\"Do you want to continue anyway?\\\"/true/g" /home/unms/app/unms-cli
+/home/unms/app/unms-cli restore-backup --file /home/unms/data/unms-backups/restore/unms-backup.unms
+```
@@ -0,0 +1,80 @@
+# UniFi Backup Restore Pitfalls (v10.0.162, Jul 2026)
+
+## The Core Problem
+
+UniFi `.unf` backup files are **encrypted BSON**. They cannot be read by:
+- `zipfile` (Python) — throws `BadZipFile`
+- `bsondump` (MongoDB) — reports "invalid BSONSize"
+- `mongorestore` — can't parse the encryption
+
+The ONLY way to restore a `.unf` is through the UniFi `ace.jar restore` command, which decrypts and imports the data into a running MongoDB.
+
+## Failed Restore Approaches
+
+### 1. CLI: `java -jar ace.jar restore <file>` (FAILED)
+
+**Error:** `NoSuchBeanDefinitionException: No qualifying bean of type 'com.ubnt.service.system.status.o0OO'`
+
+**Root cause:** `ace.jar restore` instantiates the FULL Spring application context, not just a MongoDB connection. It requires all UniFi service beans to be available. Running it in isolation (even with just MongoDB) fails because the Spring context expects the complete application environment.
+
+**Attempted workarounds that also failed:**
+- SIGSTOP the running UniFi Java process, then run restore → Spring context collision
+- SIGKILL the process → entrypoint restarts it
+- One-off Docker container with just mongod + ace.jar → same Spring bean missing
+
+### 2. API: `POST /api/s/default/cmd/backup` (FAILED)
+
+**Error:** `{"meta":{"rc":"error","msg":"api.err.NoSiteContext"}}`
+
+**Root cause:** In UniFi 10.x, even with valid authentication (`unifises` cookie from `/api/login` → `{"meta":{"rc":"ok"}}`), the backup/restore API endpoints require a site context. `/api/self` returns admin details correctly, but any site-scoped endpoint fails.
+
+**Prerequisites that were configured correctly and still failed:**
+- Admin with `super_site_permissions: ["super"]`, `is_super: true`, `super_site_role: "super"`
+- Privilege records with ObjectId `admin_id` and `site_id` for both "super" and "default" sites
+- Role: `SUPER_ADMIN`, permissions: `["ALL"]`
+
+**Attempted endpoints that all return NoSiteContext or Invalid:**
+- `/api/s/default/cmd/backup` (with `cmd: "list-backups"` or `cmd: "restore"`)
+- `/api/s/super/cmd/backup`
+- `/api/s/default/cmd/system`
+- `/api/system/backup`
+- `/api/system/backup/list`
+- `/api/system/backup/restore`
+
+## What DOES Work
+
+### Fresh deploy + re-adopt devices
+
+When the controller is deployed fresh (no backup restore), devices that were previously adopted to another controller will try to inform but fail with "inform decryption failed." To re-adopt:
+
+1. SSH into each UniFi device
+2. Run: `set-inform http://<controller-ip>:8080/inform`
+3. The device appears in the controller UI as "pending adoption"
+4. Click "Adopt" in the UI
+
+### Backup restore via the UI Setup Wizard
+
+On FIRST boot of `jacobalberty/unifi`, the setup wizard at `https://<host>:8443` offers a "Restore from Backup" option. This is the only reliable path to restore a `.unf` file on v10.x.
+
+### What to check before deploying to avoid this
+
+1. **Always export a `.unf` before decommissioning** the old controller
+2. **Store the `.unf` in S3** — don't rely on it surviving on the server
+3. **Note the old controller version** — major version jumps (9.x → 10.x) add restore risk
+4. **Document device SSH credentials** — needed for re-adoption if restore fails
+
+## Dual-Storage Architecture Trap
+
+The `jacobalberty/unifi` Docker image uses TWO storage locations that can diverge:
+
+```
+Host /opt/unifi  →  Container /config   (bind mount: system.properties, certs, old backups)
+Docker volume    →  Container /unifi    (anonymous: MongoDB data, live backups, sites)
+```
+
+The container symlinks `/usr/lib/unifi/data` → `/unifi/data`. When running a one-off container for restore, you MUST use `--volumes-from` to access the anonymous volume. Simply bind-mounting `/opt/unifi:/config` only gives you the stale config files, not the live MongoDB.
+
+To inspect the anonymous volume path:
+```bash
+docker inspect unifi-controller --format '{{json .Mounts}}' | python3 -m json.tool
+```
@@ -0,0 +1,33 @@
+#!/bin/bash
+# MongoDB init script for UniFi Network Application
+# Mount at /docker-entrypoint-initdb.d/init-mongo.sh in the mongo container.
+# Creates the 'unifi' user with proper roles on databases: unifi, unifi_stat, unifi_audit, unifi_restore
+#
+# Env vars (set on the mongo container):
+#   MONGO_INITDB_ROOT_USERNAME  - root admin user
+#   MONGO_INITDB_ROOT_PASSWORD  - root admin password
+#   MONGO_USER                  - unifi app user to create
+#   MONGO_PASS                  - unifi app user password (raw, NOT URL-encoded)
+#   MONGO_DBNAME                - base database name (stat/audit/restore are suffixed)
+#   MONGO_AUTHSOURCE            - auth database (typically 'admin')
+
+if which mongosh > /dev/null 2>&1; then
+  mongo_init_bin='mongosh'
+else
+  mongo_init_bin='mongo'
+fi
+"${mongo_init_bin}" <<EOF
+use ${MONGO_AUTHSOURCE}
+db.auth("${MONGO_INITDB_ROOT_USERNAME}", "${MONGO_INITDB_ROOT_PASSWORD}")
+db.createUser({
+  user: "${MONGO_USER}",
+  pwd: "${MONGO_PASS}",
+  roles: [
+    "clusterMonitor",
+    { db: "${MONGO_DBNAME}", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_stat", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_audit", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_restore", role: "dbOwner" }
+  ]
+})
+EOF
@@ -0,0 +1,73 @@
+# UniFi Local Network API Port Troubleshooting
+
+Use this reference when troubleshooting customer UniFi switch-port/client issues where Site Manager data is too shallow.
+
+## Key distinction
+
+- **Site Manager API** (`https://api.ui.com/v1` / `/ea` with `X-API-KEY`) gives high-level site/host/device inventory and WAN issue summaries.
+- **Local Network API** is required for clients, events, detailed switch port stats, CRC/drop counters, PoE state/actions, and port power cycling.
+
+Create local key per console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+Store per-site keys securely and label them by customer/console.
+
+## Liberty Tire identifiers found Jul 14, 2026
+
+```text
+Host: Liberty-UDM-Pro
+Host ID: D8B3702867A1000000000770A2860000000007CBEC34000000006437A55B:1003909054
+Site ID: 66cfc43699ec0d7200188ed3
+WAN/IP shown by Site Manager: 108.175.202.88
+Gateway MAC: d8:b3:70:28:67:a1
+Switch: USW-24-PoE
+Switch MAC: E4:38:83:94:66:E3
+Switch IP: 10.1.0.5
+Phone issue: Yealink W70 cordless base on switch port 1 periodically develops static/packet-loss symptoms; user's manual fix is power-cycle port 1.
+```
+
+## GLC identifiers found Jul 14, 2026
+
+```text
+Host: GLC UDM Pro
+Host ID: AC8BA96EDB4F0000000007153E2D00000000076AA09A0000000063974676:10670668
+Site ID: 66770d7ed634742ade16b848
+WAN/IP shown by Site Manager: 75.62.181.131
+```
+
+## Troubleshooting workflow
+
+1. **Read-only first**
+   - Identify switch and target port.
+   - Identify client MAC/IP/hostname (Yealink W70 base).
+   - Pull recent client history and switch port metrics.
+   - Check port link flaps, speed/duplex, PoE draw/state, RX/TX drops, CRC/FCS errors.
+   - Correlate with report time.
+2. **Check WAN separately**
+   - Site Manager may show high-latency WAN events; do not confuse those with port-level packet loss unless phone RTP paths confirm WAN jitter.
+3. **Only after approval**
+   - Execute PoE/port cycle on the specific port.
+   - Verify client returns and counters reset/stabilize.
+4. **Long-term watchdog**
+   - No-agent script checks port/client health periodically.
+   - Alert on flaps/errors/phone offline.
+   - Approval-gated action can power-cycle the known port.
+
+## Port action endpoint shape
+
+UniFi docs show:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+The exact action name/body depends on Network version; inspect the local Network app's Integrations docs for that console before writing automation.
+
+## Guardrails
+
+- Do not cycle ports without explicit user approval.
+- Confirm port numbering in API is the same as UI label before action; some APIs use zero-based `portIdx`, while UI labels are one-based.
+- Record before/after counters for any power cycle action.
@@ -0,0 +1,103 @@
+# UniFi MongoDB Admin Schema (v10.0.162)
+
+## Database: `ace`
+
+### `admin` Collection — Admin Accounts
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| `_id` | ObjectId | auto | MongoDB ID |
+| `name` | string | yes | Display name |
+| `email` | string | yes | Login email (username) |
+| `x_shadow` | string | yes | SHA-512 crypt password hash (`$6$salt$hash`) |
+| `time_created` | int32 (epoch) | yes | Account creation timestamp |
+| `last_site_name` | string | no | Last site accessed (set on login) |
+| `super_site_permissions` | array | **yes** | Must be `["super"]` for super admin |
+| `super_site_role` | string | **yes** | Must be `"super"` |
+| `is_super` | bool | **yes** | Must be `true` |
+| `last_login_timestamp` | int64 (epoch ms) | auto | Set on login |
+| `last_login_ip` | string | auto | Set on login |
+| `ui_settings` | object | auto | `{"preferredLanguage":"en"}` etc. |
+| `ui_version` | string | auto | UI version string |
+| `email_alert_enabled` | bool | no | Email notifications |
+| `is_owner` | bool | auto | Set to `true` on first admin |
+
+**Critical:** Without `super_site_permissions: ["super"]`, `super_site_role: "super"`, and `is_super: true`, the admin will log in but see an empty dashboard even if the database has devices and configs.
+
+### `privilege` Collection — Site Access Control
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| `_id` | ObjectId | auto | MongoDB ID |
+| `admin_id` | ObjectId | **yes** | References `admin._id` — MUST be ObjectId, NOT string |
+| `site_id` | ObjectId | **yes** | References `site._id` — MUST be ObjectId, NOT string |
+| `role` | string | **yes** | `"SUPER_ADMIN"` or `"ADMIN"` |
+| `permissions` | array | yes | `["ALL"]` for full access |
+| `is_super` | bool | **yes** | `true` for super admin |
+
+**Critical pitfall:** Using `.str` on ObjectIds when inserting privileges (e.g., `admin._id.str` instead of `admin._id`) creates string references that don't match. The admin logs in but sees NO data — exactly the "no content on screen" symptom.
+
+### `site` Collection
+
+| Field | Type | Description |
+|---|---|---|
+| `_id` | ObjectId | Site ID |
+| `name` | string | `"super"` or `"default"` |
+| `desc` | string | Description |
+| `attr_hidden_id` | string | Internal site key |
+| `attr_hidden` | bool | `true` for "super" site |
+| `attr_no_delete` | bool | `true` for both default sites |
+
+A fresh `jacobalberty/unifi` deploy creates two sites automatically: `"super"` (hidden) and `"default"`.
+
+### Full Admin Creation Script
+
+```javascript
+db = db.getSiblingDB("ace");
+
+// 1. Generate hash externally: openssl passwd -6 'password'
+var hash = "$6$salt$hash";
+var now = Math.floor(new Date().getTime() / 1000);
+
+// 2. Create admin
+var result = db.admin.insertOne({
+    name: "Admin Name",
+    email: "admin@example.com",
+    x_shadow: hash,
+    time_created: now,
+    last_site_name: "default",
+    super_site_permissions: ["super"],
+    super_site_role: "super",
+    is_super: true
+});
+
+// 3. Create privileges for ALL sites
+var admin = db.admin.findOne({email: "admin@example.com"});
+var sites = db.site.find({}).toArray();
+sites.forEach(function(site) {
+    db.privilege.insertOne({
+        admin_id: admin._id,       // ObjectId, NOT .str
+        site_id: site._id,         // ObjectId, NOT .str
+        role: "SUPER_ADMIN",
+        permissions: ["ALL"],
+        is_super: true
+    });
+});
+
+print("Admin created: " + admin._id);
+print("Privileges: " + db.privilege.count());
+```
+
+### Verifying Permissions After Login
+
+```bash
+# Login
+curl -sk -c /tmp/uf-cookie -X POST https://localhost:8443/api/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"admin@example.com","password":"password"}'
+
+# Check self — should show is_super:true
+curl -sk -b /tmp/uf-cookie https://localhost:8443/api/self | python3 -m json.tool
+```
+
+If `is_super` is `false` or `super_site_permissions` is `[]`, the MongoDB update didn't take — likely the admin document was overwritten by the login process or the update used the wrong query.
@@ -0,0 +1,91 @@
+# UniFi Site Manager API vs local Network API
+
+## Lesson
+
+The Site Manager API key from `api.ui.com` is useful for high-level cloud inventory, but it does not expose enough detail for switch-port diagnostics or PoE port actions.
+
+## Site Manager API can provide
+
+Endpoints such as:
+
+```text
+GET https://api.ui.com/v1/sites
+GET https://api.ui.com/ea/sites
+GET https://api.ui.com/ea/hosts
+GET https://api.ui.com/ea/devices
+```
+
+Useful for:
+
+- host/site IDs
+- gateway names and WAN IPs
+- high-level device inventory
+- online/offline counts
+- rough internet health summaries
+
+Example fields seen:
+
+```text
+siteId
+hostId
+hostName
+ipAddress
+meta.desc
+statistics.counts
+statistics.internetIssues
+```
+
+## Site Manager API was insufficient for
+
+- switch port stats
+- CRC/FCS errors
+- packet drops
+- port up/down events
+- PoE state/history
+- client history by MAC/IP
+- approval-gated PoE power cycle
+
+## Local Network API is required for port work
+
+For workflows like a Yealink W70 cordless base on a UniFi switch port periodically producing static, use the local UniFi Network API key created inside each console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+The port action endpoint documented by UniFi has this shape:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+## Liberty / GLC identifiers discovered
+
+Liberty:
+
+```text
+Host: Liberty-UDM-Pro
+Host ID: D8B3702867A1000000000770A2860000000007CBEC34000000006437A55B:1003909054
+Site ID: 66cfc43699ec0d7200188ed3
+WAN/IP: 108.175.202.88
+Switch: USW-24-PoE
+Switch MAC: E4:38:83:94:66:E3
+Switch IP: 10.1.0.5
+```
+
+GLC:
+
+```text
+Host: GLC UDM Pro
+Host ID: AC8BA96EDB4F0000000007153E2D00000000076AA09A0000000063974676:10670668
+Site ID: 66770d7ed634742ade16b848
+WAN/IP: 75.62.181.131
+```
+
+## Safe workflow for port power-cycling
+
+1. Read-only first: identify switch, port, client MAC/IP, and current counters.
+2. Correlate incident time with port errors/flaps and WAN latency.
+3. Do not power-cycle without explicit approval.
+4. If approved, execute the port action and verify link/client recovery.
+5. Keep a small incident log for repeated port/phone failures.
@@ -0,0 +1,53 @@
+# UNMS/UISP Backup/Restore — Full Sequence
+
+## Nightly Backup Location
+`/home/unms/data/unms-backups/backups/` — `YYYYMMDDHHMMSS-auto-<version>.unms` (~140MB)
+
+- old server (5.161.225.131) → `s3://hermes-vps-backups/unms-backups/old-unms/`
+- live server (152.53.39.202) → `s3://hermes-vps-backups/unms-backups/live/`
+- UCRM DB → `s3://hermes-vps-backups/unms-backups/live/ucrm/`
+
+## After Restore to New Server
+
+**Order matters:** cert → device-ws → vault
+
+### 1. Check UFW (netcup blocks all inbound except SSH)
+```bash
+ufw allow 80/tcp && ufw allow 443/tcp && ufw reload
+```
+
+### 2. Get Let's Encrypt certs
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com --preferred-challenges http
+docker start unms-nginx
+```
+
+### 3. Inject certs into UNMS Nginx
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+### 4. Start WebSocket containers
+```bash
+for i in 1 2 3 4 5 6 7 8 9 10 11; do docker start unms-device-ws-$i; done
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# All should show "Up X seconds (healthy)"
+```
+
+Without this, devices fail with: `connection error (host:443): HS: ws upgrade response not 101`
+
+### 5. Vault key — destroyed on restore
+Not recoverable from filesystem. Destroy via UI:
+Settings → Credentials → Vault → Destroy Vault
+
+### 6. Device connection string format
+```
+wss://unms.forefrontwireless.com:443+<token>+allowSelfSignedCertificate
+```