Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,569 @@
+---
+name: ubiquiti-controller-management
+description: Umbrella for Ubiquiti UISP (UNMS) and UniFi Network controller deployment, migration, and backup operations.
+---
+
+This skill covers the installation, migration, and maintenance of Ubiquiti infrastructure controllers (UISP/UNMS and UniFi).
+
+## UniFi Admin Password Reset via MongoDB (Verified v10.0.162, Jul 2026)
+
+When locked out of the UniFi controller, create an admin account directly in MongoDB. The `jacobalberty/unifi` image runs MongoDB on port **27117** (not 27017), bound to 127.0.0.1. The shell binary varies by image version: **`mongo`** on older builds, **`mongosh`** on newer ones. Always check which is available:
+
+```bash
+docker exec unifi-controller which mongosh 2>/dev/null && echo 'mongosh' || echo 'mongo'
+```
+
+**Pitfall:** Some servers (e.g. Hetzner bare-metal with UniFi installed via dpkg) have `mongosh` but no `mongo` binary at all. Running `mongo` returns `No such file or directory` (exit 127) even though MongoDB is running on 27117. Use the correct shell for the server.
+
+### Steps
+
+1. **Check existing admins:**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   print("admins: " + db.admin.count());
+   print("sites: " + db.site.count());
+   '
+   ```
+
+2. **Generate a SHA-512 crypt password hash:**
+   ```bash
+   openssl passwd -6 'TempPassword123!'
+   # Output: $6$salt$hash...
+   ```
+
+3. **Insert the admin:**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   var now = Math.floor(new Date().getTime() / 1000);
+   db.admin.insertOne({
+       name: "Admin Name",
+       email: "admin@example.com",
+       x_shadow: "$6$...",
+       time_created: now,
+       last_site_name: "default",
+       super_site_permissions: ["super"],
+       super_site_role: "super",
+       is_super: true
+   });
+   '
+   ```
+
+4. **Create privilege records (REQUIRED — without these, the UI is blank):**
+   ```bash
+   docker exec unifi-controller mongo --port 27117 --quiet --eval '
+   db = db.getSiblingDB("ace");
+   var admin = db.admin.findOne({email: "admin@example.com"});
+   var sites = db.site.find({}).toArray();
+   sites.forEach(function(site) {
+       db.privilege.insertOne({
+           admin_id: admin._id,
+           site_id: site._id,
+           role: "SUPER_ADMIN",
+           permissions: ["ALL"],
+           is_super: true
+       });
+   });
+   print("Privileges created: " + db.privilege.count());
+   '
+   ```
+
+### Critical Pitfalls
+
+- **Admin `_id` and site `_id` MUST be ObjectIds, not strings** in the privilege records. Using `.str` will cause the admin to see a blank page after login.
+- **`super_site_permissions` must be `["super"]`** — an empty array means no site access.
+- **`is_super: true`** must be set on both the admin document and privilege records.
+- If the admin was created before privileges, the existing session cookie caches the old (empty) permissions. Log out and log back in.
+
+### Verification
+
+```bash
+curl -sk -c /tmp/cookie -X POST https://localhost:8443/api/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"admin@example.com","password":"TempPassword123!"}'
+# Expect: {"meta":{"rc":"ok"}}
+
+curl -sk -b /tmp/cookie https://localhost:8443/api/self
+# Expect: admin details with is_super:true
+```
+
+### How UniFi Stores Users
+
+| Collection | Purpose |
+|---|---|
+| `ace.admin` | Admin accounts (name, email, x_shadow hash) |
+| `ace.privilege` | Links admins to sites with roles |
+| `ace.site` | Sites ("super" and "default") |
+| `ace.account` | Local device/guest accounts (not admins) |
+
+See `references/unifi-mongodb-admin-schema.md` for full field reference.
+
+## UniFi Backup File Format
+
+UniFi `.unf` backup files are **encrypted BSON** — not ZIP files. Python's `zipfile` and `bsondump` both fail. The backups can **only** be decrypted and imported by the UniFi `ace.jar restore` command. There is no third-party tool to extract `.unf` contents.
+
+### Restoring via one-off container (ace.jar restore)
+
+The `ace.jar restore` command initializes a full Spring application context. It CANNOT run while the UniFi service is active — the two contexts collide. Stop the main container, then run restore in an isolated one-off that shares volumes:
+
+```bash
+# 1. Copy .unf backup into the anonymous volume
+docker cp backup.unf unifi-controller:/unifi/data/backup/autobackup/
+docker stop unifi-controller
+
+# 2. Run restore in one-off with --volumes-from
+docker run --rm --volumes-from unifi-controller \
+  --entrypoint /bin/bash jacobalberty/unifi:latest \
+  -c "
+    mongod --dbpath /unifi/data/db --port 27117 --bind_ip 127.0.0.1 --fork --logpath /tmp/mongod.log
+    sleep 3
+    java -jar /usr/lib/unifi/lib/ace.jar restore /unifi/data/backup/autobackup/backup.unf
+    mongosh --port 27117 admin --eval 'db.shutdownServer()'
+  "
+
+# 3. Start main container
+docker start unifi-controller
+```
+
+**Pitfall — quoting hell:** When running this through SSH with nested quotes, write the restore commands to a script file and mount it into the one-off container (`-v /tmp/restore.sh:/tmp/restore.sh:ro`). Shell escaping across SSH → Docker → bash cascades is fragile.
+
+**Pitfall — Spring context collision:** Without `--volumes-from`, the one-off sees no `/unifi/data/db` (the path inside the container is `/unifi/data/db`, not `/usr/lib/unifi/data/db` — the latter is a symlink created by the entrypoint). Using only `-v /opt/unifi:/config` misses the live MongoDB data entirely.
+
+See `references/unifi-backup-restore-pitfalls.md` for additional restore dead-ends encountered with v10.x.
+
+## `jacobalberty/unifi` Dual-Storage Architecture
+
+This Docker image uses TWO separate storage locations:
+
+| Mount | Host Path | Container Path | Purpose |
+|---|---|---|---|
+| Bind mount | `/opt/unifi` | `/config` | system.properties, keystore, firmware.json, backup/ |
+| Anonymous volume | Docker-managed | `/unifi` | MongoDB data (db/), backup/ (live), sites/ |
+
+**These can drift apart.** The `/unifi` anonymous volume holds the current MongoDB, while `/config` holds config files placed during deployment. After a container restart, `/unifi/data` gets fresh MongoDB while `/config` keeps the old files. The container uses `/unifi/data/` as the live data directory (symlinked from `/usr/lib/unifi/data` → `/unifi/data`).
+
+**When running a one-off restore container**, you need `--volumes-from` to access the anonymous volume, not just `-v /opt/unifi:/config`. The data path inside the container for mongod is `/unifi/data/db`, not `/usr/lib/unifi/data/db`.
+
+## UniFi Site Manager vs Local Network API
+
+The UniFi Site Manager API (`https://api.ui.com/v1` or `/ea`) is useful for high-level inventory only: sites, hosts, device summaries, WAN issue summaries. It does **not** expose enough detail for switch-port troubleshooting or PoE actions: port counters, client history, events, PoE cycle, CRC/drop data.
+
+For port-level troubleshooting or automation on UDM/UDM-Pro sites, create a **local Network API key** inside each console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+Then use the console-local Network API for clients/devices/ports/actions. The port action endpoint shape documented by UniFi is:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+Use read-only checks first. Do not power-cycle PoE or ports without explicit user approval. See `references/unifi-local-network-api-port-troubleshooting.md` for the Liberty/GLC identifiers and Yealink W70 workflow pattern.
+
+## UniFi Network Controller Deployment
+
+### PREFERRED: `jacobalberty/unifi:latest` (verified Jul 12, 2026 on app2)
+
+**This is the only image that works reliably.** Bundles MongoDB internally — single `docker run`, no external DB, no auth setup. First boot ~30s to HTTP 200 on 8443. Container healthcheck works out of box.
+
+### Simple Deployment (app2 reference — verified working)
+
+```bash
+docker run -d --restart unless-stopped \
+  --name unifi-controller \
+  -p 8443:8443 \
+  -p 8080:8080 \
+  -p 8880:8880 \
+  -p 8843:8843 \
+  -p 3478:3478/udp \
+  -p 10001:10001/udp \
+  -v /opt/unifi:/config \
+  -e TZ=America/New_York \
+  -e RUNAS_UID0=true \
+  jacobalberty/unifi:latest
+```
+
+Wait 30-60 seconds for first boot, then access at `https://<host>:8443`. Import a `.unf` backup file through the setup wizard to restore from an old controller.
+
+### ALTERNATIVE (complex, NOT recommended): `linuxserver/unifi-network-application`
+
+**Requires external MongoDB 7.0 with auth.** This image does NOT bundle MongoDB internally.
+Bundles MongoDB internally — single `docker run`, no external DB, no auth setup.
+- First boot ~30s to HTTP 200 on 8443
+- Container healthcheck works out of box
+- **Verified working on app2 (152.53.39.202) Jul 12 2026**
+
+**Option B: `linuxserver/unifi-network-application` (external MongoDB REQUIRED)**
+This image does NOT bundle MongoDB. Must provide external MongoDB 7.0 with auth.
+
+**CRITICAL PITFALL with Option B:** The `linuxserver/unifi-network-application` image has a confirmed bug where `MONGO_USER` and `MONGO_PASS` environment variables are NOT honored by the Java application. Even with correctly configured MongoDB auth and all env vars set (`MONGO_USER`, `MONGO_PASS`, `MONGO_HOST`, `MONGO_PORT`, `MONGO_DBNAME`, even `MONGO_URI`), the controller throws:
+```
+java.lang.IllegalArgumentException: No username is provided in the connection string
+```
+This was tested with: MongoDB 7.0 with auth, MongoDB 7.0 without auth, explicit `MONGO_URI=mongodb://user:pass@host:27017/db`, and the individual `MONGO_USER`/`MONGO_PASS` env vars. None worked.
+
+**Fix:** Switch to Option A (`jacobalberty/unifi`). It bundles MongoDB with the correct auth setup and works on first boot. The linuxserver image's Java connection string construction ignores the user/pass env vars, making it unusable for any deployment that requires authentication.
+
+### MongoDB Requirement
+
+Starting with version 8.1, UniFi Network Application supports MongoDB 3.6 through 7.0. Version 9.0+ also supports MongoDB 8.0. Pin your MongoDB version — do NOT use `latest`, as MongoDB does not support automatic upgrades between major versions. Use `mongo:7.0` for broad compatibility.
+
+### Password Encoding (Critical Pitfall)
+
+**Special characters in the MongoDB password MUST be URL-encoded** in the `MONGO_PASS` env var passed to the UniFi container. Characters that break the MongoDB connection string if not encoded: `@` → `%40`, `:` → `%3A`, `%` → `%25`, `!` → `%21`. The easiest approach is to use **alphanumeric-only passwords** (no special chars) for both the MongoDB init script and the UniFi container to avoid encoding issues entirely.
+
+When you do need special characters, encode them in `MONGO_PASS` only — the MongoDB init script uses the raw password to create the user, but the UniFi container's Java process builds a MongoDB connection string from the env var and needs it encoded.
+
+### Port Mappings
+
+| Port | Purpose | Required |
+|------|---------|----------|
+| 8443 | Web admin UI (HTTPS) | Yes |
+| 8080 | Device communication | Yes |
+| 3478/udp | STUN | Yes |
+| 10001/udp | AP discovery | Yes |
+| 8880 | Guest portal HTTP redirect | Optional |
+| 8843 | Guest portal HTTPS redirect | Optional |
+| 1900/udp | L2 controller discoverability | Optional |
+| 6789 | Mobile throughput test | Optional |
+| 5514/udp | Remote syslog | Optional |
+
+**Do NOT use ports 443 or 80** — the UniFi controller does not need them, and they commonly conflict with Caddy or other web services. The controller uses self-signed TLS on 8443; device inform uses HTTP on 8080.
+
+### Access and First-Run Wizard
+
+The web UI is at `https://<host>:8443`. On first run, use the setup wizard to either configure fresh or restore a backup. UniFi takes 2-3 minutes to fully initialize on first launch (MongoDB init + Java application startup). Port 8443 will be listening before the app is ready; curl will get TLS errors until initialization completes.
+
+Monitor progress with: `docker logs -f unifi-controller`
+
+### Post-Deployment Verification
+
+```bash
+docker ps --filter name=unifi --format 'table {{.Names}}\\t{{.Status}}'
+curl -sk https://localhost:8443/ | head -5
+```
+
+**Pitfall — API returns HTML during initialization:** The `/api/login` and all other API endpoints return the "Network application is starting up..." HTML status page (not JSON) while UniFi initializes. Curl calls to any API endpoint will get full HTML pages during this window. The `/status` endpoint is the authoritative check — wait for `\"up\":true` before calling any API. On first boot this is 30-60 seconds; on container restart, 5-15 seconds. Loop until ready:
+
+## Caddy Reverse Proxy for UniFi (Self-Signed Cert)
+
+When proxying UniFi behind Caddy with a public domain (e.g., `unifi.itpropartner.com` → `localhost:8443`), UniFi's self-signed TLS cert requires `tls_insecure_skip_verify` and HTTP transport mode:
+
+```caddy
+unifi.itpropartner.com {
+    reverse_proxy localhost:8443 {
+        transport http {
+            tls_insecure_skip_verify
+        }
+    }
+}
+```
+
+**Without `tls_insecure_skip_verify`**, Caddy returns `TLS alert, internal error` when trying to connect to UniFi's self-signed cert. The **browser sees a blank page** — no error, no content, just nothing.
+
+**With the correct config**, Caddy terminates public TLS (Let's Encrypt), then proxies HTTP to UniFi on 8443 with cert verification disabled. The browser gets valid HTTPS.
+
+**Pitfall:** `curl -H 'Host: unifi.itpropartner.com' https://<server-ip>/` tests Caddy's routing but NOT SNI. Use the actual domain to test: `curl -skL https://unifi.itpropartner.com/`. When the proxy works, you'll get `HTTP/2 302 → /manage` followed by the UniFi SPA HTML.
+
+### Access and First-Run Wizard
+
+The web UI is at `https://<host>:8443`. On first run, use the setup wizard to either configure fresh or restore a backup. UniFi takes 2-3 minutes to fully initialize on first launch (MongoDB init + Java application startup). Port 8443 will be listening before the app is ready; curl will get TLS errors until initialization completes.
+
+Monitor progress with: `docker logs -f unifi-controller`
+
+## UNMS/UISP Backup to S3 — Dual-Source Archive (updated Jul 12, 2026)
+
+Syncs backup from BOTH the old server (historical archive) AND the live server (current operations) to S3.
+
+**Old server (historical):** `5.161.225.131` → `s3://hermes-vps-backups/unms-backups/old-unms/`
+**Live server (current):** `152.53.39.202` (app2) → `s3://hermes-vps-backups/unms-backups/live/`
+
+**Backup location on both servers:** `/home/unms/data/unms-backups/backups/`
+**Backup files:** Named `YYYYMMDDHHMMSS-auto-<version>.unms`, ~140MB each
+**Backup frequency:** Nightly auto-backup around 4:00 AM
+**UCRM backup also at:** `/home/unms/data/ucrm/ucrm/data/backup/`
+
+**S3 sync script pattern (saved at /root/.hermes/scripts/unms-backup-sync.sh):**
+
+Do **not** run `rsync` directly to an `s3://...` URL. `rsync` does not speak S3; that pattern silently fails or copies nothing useful. Use a two-step pipeline:
+
+1. Install `rsync` on both the agent host and the remote UISP/UNMS host.
+2. `rsync` remote backup directories to local staging under root disk, not `/tmp`.
+3. Upload staging to Wasabi with `aws s3 sync --endpoint-url https://s3.us-east-1.wasabisys.com`.
+4. Verify S3 has at least one object under the live backup prefix before returning success.
+
+```bash
+STAGE="/root/.hermes/.backups/unms-sync"
+S3_BUCKET="s3://hermes-vps-backups/unms-backups"
+ENDPOINT="https://s3.us-east-1.wasabisys.com"
+SSH_KEY="/root/.ssh/itpp-infra"
+
+rsync -az --delete -e "ssh -i ${SSH_KEY} -o BatchMode=yes" \
+  root@152.53.39.202:/home/unms/data/unms-backups/backups/ \
+  "$STAGE/live/backups/"
+
+aws s3 sync "$STAGE/" "$S3_BUCKET/" --endpoint-url "$ENDPOINT" --delete
+aws s3 ls "$S3_BUCKET/live/backups/" --endpoint-url "$ENDPOINT" | grep .
+```
+
+**Cron schedule:** Daily at 6 AM ET, no_agent mode, script-only.
+Created via: `cronjob(action='create', name='unms-backup-sync', schedule='0 6 * * *', script='unms-backup-sync.sh', no_agent=True)`
+
+**Verification:** Check S3 bucket has files after first run:
+```bash
+aws s3 ls --recursive s3://hermes-vps-backups/unms-backups/ \
+  --endpoint-url https://s3.us-east-1.wasabisys.com | tail -5
+```
+
+**This captures 3 data streams:**
+1. Full UISP .unms auto-backups (auto every ~24h, manual download also works)
+2. UCRM database backups (found at `/home/unms/data/ucrm/ucrm/data/backup/database/`)
+3. Config snapshots (UUID-labeled directories at `/home/unms/data/config-backups/`)
+
+## UNMS Nginx Port Architecture (Critical for Reverse Proxies)
+
+The UNMS nginx container (`ubnt/unms-nginx`) listens on multiple ports, each with a distinct purpose. **Misrouting web traffic to the wrong port causes cryptic errors.**
+
+| Port | Protocol | Purpose | Proxy-Ready |
+|------|----------|---------|-------------|
+| 80 (internal) | HTTP | ACME challenges only; redirects to 443 | ❌ Redirect loop |
+| 443 (internal) | HTTPS | **Main web UI** — `/nms/login`, full UNMS/UCRM | ✅ **Proxy here** |
+| 81 | HTTP | UCRM suspend page | ❌ 503 for `/nms/*` |
+| 8089 | HTTPS (SSL) | **WebSocket tunnel** (`REMOTE_UI_WS_PORT_PUBLIC`) — always sends `Upgrade` header to `unms-api:8085` | ❌ **NEVER proxy web traffic here** |
+
+### The Port 8089 Trap
+
+Port 8089 is often the only HTTPS port exposed from the nginx container (docker-compose default: only `81:81` and `8089:8089`). It's tempting to point a reverse proxy at it. **Don't.** This port:
+
+- Is `listen 8089 ssl` — expects TLS from the client
+- Has **one location block** that ALWAYS sets `proxy_set_header Upgrade` and `Connection: upgrade`
+- Proxies to `unms-api:8085` (WebSocket endpoint, not the web UI)
+- Returns `"Upgrade Required"` (426) for regular HTTP requests
+- Returns `"400 The plain HTTP request was sent to HTTPS port"` if you send plain HTTP to it
+
+**Symptoms of misrouting to 8089:**
+- `curl https://unms.forefrontwireless.com/nms/login` → HTTP 426 "Upgrade Required"
+- `curl http://localhost:8089` → HTTP 400 "The plain HTTP request was sent to HTTPS port" (from OpenResty)
+
+### Exposing the Real Web UI (Port 443)
+
+The UNMS web UI lives on the container's **internal port 443**, which is NOT exposed by default. Add it to the nginx service in `/home/unms/app/docker-compose.yml`:
+
+```yaml
+nginx:
+  ports:
+    - 81:81
+    - 8089:8089
+    - 8444:443    # ← ADD THIS (use port that doesn't conflict)
+```
+
+**Port selection:** On app2, 8443 is taken by UniFi. Use 8444 or another free port. Check with `ss -tlnp | grep 844` before choosing.
+
+After editing the compose file, recreate the container:
+
+```bash
+docker stop unms-nginx && docker rm unms-nginx
+docker compose -p unms -f /home/unms/app/docker-compose.yml up -d nginx
+```
+
+**Pitfall — compose project name mismatch:** The UNMS installer creates containers with project name `unms` (networks: `unms_public`, `unms_internal`), but the compose file lives in `/home/unms/app/`. Running `docker compose` without `-p unms` defaults to project `app`, which tries to create `app_public`/`app_internal` networks and fails with `invalid pool request: Pool overlaps with other one on this address space`. **Always use `docker compose -p unms`** when managing UNMS services.
+
+### Caddy Reverse Proxy for UNMS (Self-Signed Cert)
+
+UNMS uses a self-signed or auto-generated cert on its internal nginx. Same pattern as UniFi — Caddy terminates public TLS, then proxies via HTTPS with `tls_insecure_skip_verify`:
+
+```caddy
+unms.forefrontwireless.com {
+    reverse_proxy https://localhost:8444 {
+        transport http {
+            tls_insecure_skip_verify
+        }
+    }
+}
+```
+
+**Verification:**
+```bash
+# Backend reachable:
+curl -sk -o /dev/null -w '%{http_code}' https://localhost:8444/nms/login
+# → 200
+
+# Public URL works:
+curl -sk -o /dev/null -w '%{http_code}' https://unms.forefrontwireless.com/nms/login
+# → 200
+```
+
+## netcup Firewall: Ports 80/443 Blocked by UFW
+
+**netcup RS-series servers ship with UFW enabled and only port 22 open by default.** This blocks all inbound HTTP/HTTPS traffic, making UNMS/UISP unreachable from the internet despite Docker containers being configured correctly.
+
+### Symptom
+- UNMS Nginx Docker container is running and listening on ports 80/443 internally
+- `curl http://localhost:80` works from the server itself
+- `curl http://<public-ip>:80` times out from any external host
+- Certbot HTTP challenge fails with "Timeout during connect (likely firewall problem)"
+- Browser shows ERR_CONNECTION_TIMED_OUT or SSL errors
+
+### Diagnosis
+```bash
+ufw status
+# Output shows only: 22/tcp ALLOW Anywhere
+ufw status verbose
+# Shows default policy: DROP on incoming
+```
+
+### Fix
+```bash
+ufw allow 80/tcp comment 'UNMS HTTP'
+ufw allow 443/tcp comment 'UNMS HTTPS'
+ufw reload
+# Verify:
+ufw status | head -10
+```
+
+### After Opening Ports
+1. UNMS Nginx becomes accessible from the internet
+2. Certbot's HTTP challenge succeeds (no port 80 conflict)
+3. Let's Encrypt certificates can be obtained for `unms.forefrontwireless.com` and `unifi.itpropartner.com`
+4. The UNMS Nginx must be stopped briefly for certbot standalone mode:
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com \
+  --preferred-challenges http
+docker start unms-nginx
+```
+
+### Certificate Injection into UNMS
+Certbot stores certs at `/etc/letsencrypt/live/<domain>/`. Inject them into UNMS:
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+Certbot auto-renewal via systemd timer is set up by default on Debian 13.
+
+### All netcup servers May Have This Issue
+This may apply to ALL netcup RS-series VPS, not just app2. When deploying any web service on a new netcup box, check UFW immediately:
+```bash
+ufw status | grep -q '80.*ALLOW' || echo "Port 80 blocked!"
+```
+
+### UNMS/UISP Backup Restore — WebSocket Container Recovery (Jul 12, 2026)
+
+After restoring a `.unms` backup to a new server, the UNMS `device-ws` containers (which handle WebSocket connections from devices) may be in `Created` state but NOT `Running`. This causes devices to fail registration with:
+
+```
+connection error (host:443): HS: ws upgrade response not 101
+```
+
+The device connects to UNMS on port 443, but since the WebSocket handler containers aren't running, the Nginx returns a regular HTTP response instead of a WebSocket upgrade (101). The device error log shows 10 connection attempts, all failing with "ws upgrade response not 101".
+
+**Diagnosis:**
+```bash
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# Output: unms-device-ws-1 Created  (NOT "Up" — not running)
+```
+
+**Fix — start all device-ws containers:**
+```bash
+for i in 1 2 3 4 5 6 7 8 9 10 11; do
+  docker start unms-device-ws-$i
+done
+# Verify all healthy:
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# All should show "Up X seconds (healthy)"
+```
+
+**After starting, re-register the device with the connection string.**
+
+### UNMS/UISP Backup Restore — Vault Key Missing
+
+After restoring a backup to a new server, the Credentials Vault key from the original installation is required. This key was set once during initial UISP setup and is NOT stored in any config file, .env, or database. It is recoverable only if the original operator wrote it down.
+
+**Error:** "Vault Key Is Missing" when accessing device credentials in the UISP UI.
+
+**Options:**
+1. **Destroy vault → create new** — UISP will auto-generate new passwords for online devices. Offline devices need manual updates. Use UISP web UI: Settings → Credentials → Vault → Destroy Vault.
+2. **Use old server as reference** — Keep the old UISP server running simultaneously while migrating, so device configs (IP, radios, names) can be looked up during manual re-registration.
+
+**Destroy path is the intended recovery mechanism** per UISP documentation: https://help.ui.com/hc/en-us/articles/360019506834
+
+### Certificate Injection After Backup Restore
+
+After restoring a backup to a new server, the UNMS Nginx will use self-signed or localhost certificates from the restored data. These must be replaced with valid Let's Encrypt certs.
+
+**Prerequisites:** UFW must have ports 80 and 443 open (see netcup UFW section above).
+
+**Step 1 — Get Let's Encrypt certs:**
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com \
+  --preferred-challenges http
+docker start unms-nginx
+```
+
+**Step 2 — Inject certs into UNMS:**
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+**Step 3 — Verify:**
+```bash
+curl -sI --connect-timeout 5 https://unms.forefrontwireless.com/nms/login | head -3
+# Expected: HTTP/2 200, x-unms-login-screen: 1
+```
+
+Certbot auto-renewal runs daily via systemd timer on Debian 13.
+
+### UNMS/UISP Backup Restore — Full Recovery Sequence (Jul 12, 2026)
+
+When migrating UNMS to a new server, the complete sequence is:
+
+1. **Prerequisite:** UFW ports 80/443 open on the new server
+2. Obtain Let's Encrypt cert (standalone mode, brief nginx stop)
+3. Inject cert into /home/unms/data/cert/{live.crt,live.key}
+4. Start all device-ws containers (they may be Created but not Running)
+5. Accept the vault key is lost — destroy and recreate via web UI
+6. Import your .unms backup through the UISP admin UI
+7. Re-register devices manually with the new connection string
+8. Set up daily S3 backup sync (cron, 6 AM ET)
+
+### Database Backup & Restore
+* **Do NOT use `pg_dump`**: The live `unms-postgres` container uses TimescaleDB with complex hypertable/chunk circular foreign keys. A standard `pg_dump` will fail to restore.
+* **Use Native Backups**: Always use the native `.unms` auto-backups located at `/home/unms/data/unms-backups/backups/`.
+* **Restore path**: Place the backup file in `/home/unms/data/unms-backups/restore/` and run `/home/unms/app/unms-cli restore-backup --file <path>`.
+
+### Docker Version Pinning Pitfalls
+The official installer (`https://uisp.ui.com/v1/install`) has strict, hardcoded Docker version requirements (e.g., demanding 27.5.1 and failing on 29.6.1). This breaks automated unattended installs and interactive restores.
+
+**To deploy on a server with a newer Docker version:**
+1. Download the installer: `curl -fsSL https://uisp.ui.com/v1/install > /tmp/uisp_inst.sh && chmod +x /tmp/uisp_inst.sh`
+2. Run it once so it downloads and unpacks `/tmp/unms-install/install-full.sh` (it will fail the version check).
+3. Patch the unpacked installer to bypass checks:
+   ```bash
+   sed -i 's/SKIP_DOCKER_INSTALL="false"/SKIP_DOCKER_INSTALL="true"/g' /tmp/unms-install/install-full.sh
+   sed -i 's/if ! is_desired_docker_version_installed; then/if false; then/g' /tmp/unms-install/install-full.sh
+   cd /tmp/unms-install && ./install-full.sh --unattended
+   ```
+
+**To restore a backup bypassing the `unms-cli` version prompt:**
+`unms-cli restore-backup` will also interactively block on the Docker version mismatch. Bypass it by patching the CLI script before restoring:
+```bash
+sed -i "s/check_docker_version/true/g" /home/unms/app/unms-cli
+sed -i "s/confirm \\\"Do you want to continue anyway?\\\"/true/g" /home/unms/app/unms-cli
+/home/unms/app/unms-cli restore-backup --file /home/unms/data/unms-backups/restore/unms-backup.unms
+```