Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,80 @@
+# UniFi Backup Restore Pitfalls (v10.0.162, Jul 2026)
+
+## The Core Problem
+
+UniFi `.unf` backup files are **encrypted BSON**. They cannot be read by:
+- `zipfile` (Python) — throws `BadZipFile`
+- `bsondump` (MongoDB) — reports "invalid BSONSize"
+- `mongorestore` — can't parse the encryption
+
+The ONLY way to restore a `.unf` is through the UniFi `ace.jar restore` command, which decrypts and imports the data into a running MongoDB.
+
+## Failed Restore Approaches
+
+### 1. CLI: `java -jar ace.jar restore <file>` (FAILED)
+
+**Error:** `NoSuchBeanDefinitionException: No qualifying bean of type 'com.ubnt.service.system.status.o0OO'`
+
+**Root cause:** `ace.jar restore` instantiates the FULL Spring application context, not just a MongoDB connection. It requires all UniFi service beans to be available. Running it in isolation (even with just MongoDB) fails because the Spring context expects the complete application environment.
+
+**Attempted workarounds that also failed:**
+- SIGSTOP the running UniFi Java process, then run restore → Spring context collision
+- SIGKILL the process → entrypoint restarts it
+- One-off Docker container with just mongod + ace.jar → same Spring bean missing
+
+### 2. API: `POST /api/s/default/cmd/backup` (FAILED)
+
+**Error:** `{"meta":{"rc":"error","msg":"api.err.NoSiteContext"}}`
+
+**Root cause:** In UniFi 10.x, even with valid authentication (`unifises` cookie from `/api/login` → `{"meta":{"rc":"ok"}}`), the backup/restore API endpoints require a site context. `/api/self` returns admin details correctly, but any site-scoped endpoint fails.
+
+**Prerequisites that were configured correctly and still failed:**
+- Admin with `super_site_permissions: ["super"]`, `is_super: true`, `super_site_role: "super"`
+- Privilege records with ObjectId `admin_id` and `site_id` for both "super" and "default" sites
+- Role: `SUPER_ADMIN`, permissions: `["ALL"]`
+
+**Attempted endpoints that all return NoSiteContext or Invalid:**
+- `/api/s/default/cmd/backup` (with `cmd: "list-backups"` or `cmd: "restore"`)
+- `/api/s/super/cmd/backup`
+- `/api/s/default/cmd/system`
+- `/api/system/backup`
+- `/api/system/backup/list`
+- `/api/system/backup/restore`
+
+## What DOES Work
+
+### Fresh deploy + re-adopt devices
+
+When the controller is deployed fresh (no backup restore), devices that were previously adopted to another controller will try to inform but fail with "inform decryption failed." To re-adopt:
+
+1. SSH into each UniFi device
+2. Run: `set-inform http://<controller-ip>:8080/inform`
+3. The device appears in the controller UI as "pending adoption"
+4. Click "Adopt" in the UI
+
+### Backup restore via the UI Setup Wizard
+
+On FIRST boot of `jacobalberty/unifi`, the setup wizard at `https://<host>:8443` offers a "Restore from Backup" option. This is the only reliable path to restore a `.unf` file on v10.x.
+
+### What to check before deploying to avoid this
+
+1. **Always export a `.unf` before decommissioning** the old controller
+2. **Store the `.unf` in S3** — don't rely on it surviving on the server
+3. **Note the old controller version** — major version jumps (9.x → 10.x) add restore risk
+4. **Document device SSH credentials** — needed for re-adoption if restore fails
+
+## Dual-Storage Architecture Trap
+
+The `jacobalberty/unifi` Docker image uses TWO storage locations that can diverge:
+
+```
+Host /opt/unifi  →  Container /config   (bind mount: system.properties, certs, old backups)
+Docker volume    →  Container /unifi    (anonymous: MongoDB data, live backups, sites)
+```
+
+The container symlinks `/usr/lib/unifi/data` → `/unifi/data`. When running a one-off container for restore, you MUST use `--volumes-from` to access the anonymous volume. Simply bind-mounting `/opt/unifi:/config` only gives you the stale config files, not the live MongoDB.
+
+To inspect the anonymous volume path:
+```bash
+docker inspect unifi-controller --format '{{json .Mounts}}' | python3 -m json.tool
+```
@@ -0,0 +1,33 @@
+#!/bin/bash
+# MongoDB init script for UniFi Network Application
+# Mount at /docker-entrypoint-initdb.d/init-mongo.sh in the mongo container.
+# Creates the 'unifi' user with proper roles on databases: unifi, unifi_stat, unifi_audit, unifi_restore
+#
+# Env vars (set on the mongo container):
+#   MONGO_INITDB_ROOT_USERNAME  - root admin user
+#   MONGO_INITDB_ROOT_PASSWORD  - root admin password
+#   MONGO_USER                  - unifi app user to create
+#   MONGO_PASS                  - unifi app user password (raw, NOT URL-encoded)
+#   MONGO_DBNAME                - base database name (stat/audit/restore are suffixed)
+#   MONGO_AUTHSOURCE            - auth database (typically 'admin')
+
+if which mongosh > /dev/null 2>&1; then
+  mongo_init_bin='mongosh'
+else
+  mongo_init_bin='mongo'
+fi
+"${mongo_init_bin}" <<EOF
+use ${MONGO_AUTHSOURCE}
+db.auth("${MONGO_INITDB_ROOT_USERNAME}", "${MONGO_INITDB_ROOT_PASSWORD}")
+db.createUser({
+  user: "${MONGO_USER}",
+  pwd: "${MONGO_PASS}",
+  roles: [
+    "clusterMonitor",
+    { db: "${MONGO_DBNAME}", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_stat", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_audit", role: "dbOwner" },
+    { db: "${MONGO_DBNAME}_restore", role: "dbOwner" }
+  ]
+})
+EOF
@@ -0,0 +1,73 @@
+# UniFi Local Network API Port Troubleshooting
+
+Use this reference when troubleshooting customer UniFi switch-port/client issues where Site Manager data is too shallow.
+
+## Key distinction
+
+- **Site Manager API** (`https://api.ui.com/v1` / `/ea` with `X-API-KEY`) gives high-level site/host/device inventory and WAN issue summaries.
+- **Local Network API** is required for clients, events, detailed switch port stats, CRC/drop counters, PoE state/actions, and port power cycling.
+
+Create local key per console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+Store per-site keys securely and label them by customer/console.
+
+## Liberty Tire identifiers found Jul 14, 2026
+
+```text
+Host: Liberty-UDM-Pro
+Host ID: D8B3702867A1000000000770A2860000000007CBEC34000000006437A55B:1003909054
+Site ID: 66cfc43699ec0d7200188ed3
+WAN/IP shown by Site Manager: 108.175.202.88
+Gateway MAC: d8:b3:70:28:67:a1
+Switch: USW-24-PoE
+Switch MAC: E4:38:83:94:66:E3
+Switch IP: 10.1.0.5
+Phone issue: Yealink W70 cordless base on switch port 1 periodically develops static/packet-loss symptoms; user's manual fix is power-cycle port 1.
+```
+
+## GLC identifiers found Jul 14, 2026
+
+```text
+Host: GLC UDM Pro
+Host ID: AC8BA96EDB4F0000000007153E2D00000000076AA09A0000000063974676:10670668
+Site ID: 66770d7ed634742ade16b848
+WAN/IP shown by Site Manager: 75.62.181.131
+```
+
+## Troubleshooting workflow
+
+1. **Read-only first**
+   - Identify switch and target port.
+   - Identify client MAC/IP/hostname (Yealink W70 base).
+   - Pull recent client history and switch port metrics.
+   - Check port link flaps, speed/duplex, PoE draw/state, RX/TX drops, CRC/FCS errors.
+   - Correlate with report time.
+2. **Check WAN separately**
+   - Site Manager may show high-latency WAN events; do not confuse those with port-level packet loss unless phone RTP paths confirm WAN jitter.
+3. **Only after approval**
+   - Execute PoE/port cycle on the specific port.
+   - Verify client returns and counters reset/stabilize.
+4. **Long-term watchdog**
+   - No-agent script checks port/client health periodically.
+   - Alert on flaps/errors/phone offline.
+   - Approval-gated action can power-cycle the known port.
+
+## Port action endpoint shape
+
+UniFi docs show:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+The exact action name/body depends on Network version; inspect the local Network app's Integrations docs for that console before writing automation.
+
+## Guardrails
+
+- Do not cycle ports without explicit user approval.
+- Confirm port numbering in API is the same as UI label before action; some APIs use zero-based `portIdx`, while UI labels are one-based.
+- Record before/after counters for any power cycle action.
@@ -0,0 +1,103 @@
+# UniFi MongoDB Admin Schema (v10.0.162)
+
+## Database: `ace`
+
+### `admin` Collection — Admin Accounts
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| `_id` | ObjectId | auto | MongoDB ID |
+| `name` | string | yes | Display name |
+| `email` | string | yes | Login email (username) |
+| `x_shadow` | string | yes | SHA-512 crypt password hash (`$6$salt$hash`) |
+| `time_created` | int32 (epoch) | yes | Account creation timestamp |
+| `last_site_name` | string | no | Last site accessed (set on login) |
+| `super_site_permissions` | array | **yes** | Must be `["super"]` for super admin |
+| `super_site_role` | string | **yes** | Must be `"super"` |
+| `is_super` | bool | **yes** | Must be `true` |
+| `last_login_timestamp` | int64 (epoch ms) | auto | Set on login |
+| `last_login_ip` | string | auto | Set on login |
+| `ui_settings` | object | auto | `{"preferredLanguage":"en"}` etc. |
+| `ui_version` | string | auto | UI version string |
+| `email_alert_enabled` | bool | no | Email notifications |
+| `is_owner` | bool | auto | Set to `true` on first admin |
+
+**Critical:** Without `super_site_permissions: ["super"]`, `super_site_role: "super"`, and `is_super: true`, the admin will log in but see an empty dashboard even if the database has devices and configs.
+
+### `privilege` Collection — Site Access Control
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| `_id` | ObjectId | auto | MongoDB ID |
+| `admin_id` | ObjectId | **yes** | References `admin._id` — MUST be ObjectId, NOT string |
+| `site_id` | ObjectId | **yes** | References `site._id` — MUST be ObjectId, NOT string |
+| `role` | string | **yes** | `"SUPER_ADMIN"` or `"ADMIN"` |
+| `permissions` | array | yes | `["ALL"]` for full access |
+| `is_super` | bool | **yes** | `true` for super admin |
+
+**Critical pitfall:** Using `.str` on ObjectIds when inserting privileges (e.g., `admin._id.str` instead of `admin._id`) creates string references that don't match. The admin logs in but sees NO data — exactly the "no content on screen" symptom.
+
+### `site` Collection
+
+| Field | Type | Description |
+|---|---|---|
+| `_id` | ObjectId | Site ID |
+| `name` | string | `"super"` or `"default"` |
+| `desc` | string | Description |
+| `attr_hidden_id` | string | Internal site key |
+| `attr_hidden` | bool | `true` for "super" site |
+| `attr_no_delete` | bool | `true` for both default sites |
+
+A fresh `jacobalberty/unifi` deploy creates two sites automatically: `"super"` (hidden) and `"default"`.
+
+### Full Admin Creation Script
+
+```javascript
+db = db.getSiblingDB("ace");
+
+// 1. Generate hash externally: openssl passwd -6 'password'
+var hash = "$6$salt$hash";
+var now = Math.floor(new Date().getTime() / 1000);
+
+// 2. Create admin
+var result = db.admin.insertOne({
+    name: "Admin Name",
+    email: "admin@example.com",
+    x_shadow: hash,
+    time_created: now,
+    last_site_name: "default",
+    super_site_permissions: ["super"],
+    super_site_role: "super",
+    is_super: true
+});
+
+// 3. Create privileges for ALL sites
+var admin = db.admin.findOne({email: "admin@example.com"});
+var sites = db.site.find({}).toArray();
+sites.forEach(function(site) {
+    db.privilege.insertOne({
+        admin_id: admin._id,       // ObjectId, NOT .str
+        site_id: site._id,         // ObjectId, NOT .str
+        role: "SUPER_ADMIN",
+        permissions: ["ALL"],
+        is_super: true
+    });
+});
+
+print("Admin created: " + admin._id);
+print("Privileges: " + db.privilege.count());
+```
+
+### Verifying Permissions After Login
+
+```bash
+# Login
+curl -sk -c /tmp/uf-cookie -X POST https://localhost:8443/api/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"admin@example.com","password":"password"}'
+
+# Check self — should show is_super:true
+curl -sk -b /tmp/uf-cookie https://localhost:8443/api/self | python3 -m json.tool
+```
+
+If `is_super` is `false` or `super_site_permissions` is `[]`, the MongoDB update didn't take — likely the admin document was overwritten by the login process or the update used the wrong query.
@@ -0,0 +1,91 @@
+# UniFi Site Manager API vs local Network API
+
+## Lesson
+
+The Site Manager API key from `api.ui.com` is useful for high-level cloud inventory, but it does not expose enough detail for switch-port diagnostics or PoE port actions.
+
+## Site Manager API can provide
+
+Endpoints such as:
+
+```text
+GET https://api.ui.com/v1/sites
+GET https://api.ui.com/ea/sites
+GET https://api.ui.com/ea/hosts
+GET https://api.ui.com/ea/devices
+```
+
+Useful for:
+
+- host/site IDs
+- gateway names and WAN IPs
+- high-level device inventory
+- online/offline counts
+- rough internet health summaries
+
+Example fields seen:
+
+```text
+siteId
+hostId
+hostName
+ipAddress
+meta.desc
+statistics.counts
+statistics.internetIssues
+```
+
+## Site Manager API was insufficient for
+
+- switch port stats
+- CRC/FCS errors
+- packet drops
+- port up/down events
+- PoE state/history
+- client history by MAC/IP
+- approval-gated PoE power cycle
+
+## Local Network API is required for port work
+
+For workflows like a Yealink W70 cordless base on a UniFi switch port periodically producing static, use the local UniFi Network API key created inside each console:
+
+```text
+UniFi Network → Integrations → Network API → Create API Key
+```
+
+The port action endpoint documented by UniFi has this shape:
+
+```text
+POST /v1/sites/{siteId}/devices/{deviceId}/interfaces/ports/{portIdx}/actions
+```
+
+## Liberty / GLC identifiers discovered
+
+Liberty:
+
+```text
+Host: Liberty-UDM-Pro
+Host ID: D8B3702867A1000000000770A2860000000007CBEC34000000006437A55B:1003909054
+Site ID: 66cfc43699ec0d7200188ed3
+WAN/IP: 108.175.202.88
+Switch: USW-24-PoE
+Switch MAC: E4:38:83:94:66:E3
+Switch IP: 10.1.0.5
+```
+
+GLC:
+
+```text
+Host: GLC UDM Pro
+Host ID: AC8BA96EDB4F0000000007153E2D00000000076AA09A0000000063974676:10670668
+Site ID: 66770d7ed634742ade16b848
+WAN/IP: 75.62.181.131
+```
+
+## Safe workflow for port power-cycling
+
+1. Read-only first: identify switch, port, client MAC/IP, and current counters.
+2. Correlate incident time with port errors/flaps and WAN latency.
+3. Do not power-cycle without explicit approval.
+4. If approved, execute the port action and verify link/client recovery.
+5. Keep a small incident log for repeated port/phone failures.
@@ -0,0 +1,53 @@
+# UNMS/UISP Backup/Restore — Full Sequence
+
+## Nightly Backup Location
+`/home/unms/data/unms-backups/backups/` — `YYYYMMDDHHMMSS-auto-<version>.unms` (~140MB)
+
+- old server (5.161.225.131) → `s3://hermes-vps-backups/unms-backups/old-unms/`
+- live server (152.53.39.202) → `s3://hermes-vps-backups/unms-backups/live/`
+- UCRM DB → `s3://hermes-vps-backups/unms-backups/live/ucrm/`
+
+## After Restore to New Server
+
+**Order matters:** cert → device-ws → vault
+
+### 1. Check UFW (netcup blocks all inbound except SSH)
+```bash
+ufw allow 80/tcp && ufw allow 443/tcp && ufw reload
+```
+
+### 2. Get Let's Encrypt certs
+```bash
+docker stop unms-nginx
+certbot certonly --standalone --non-interactive --agree-tos \
+  -m info@itpropartner.com \
+  -d unms.forefrontwireless.com --preferred-challenges http
+docker start unms-nginx
+```
+
+### 3. Inject certs into UNMS Nginx
+```bash
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
+cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
+chown unms:unms /home/unms/data/cert/live.*
+chmod 600 /home/unms/data/cert/live.key
+docker restart unms-nginx
+```
+
+### 4. Start WebSocket containers
+```bash
+for i in 1 2 3 4 5 6 7 8 9 10 11; do docker start unms-device-ws-$i; done
+docker ps --format '{{.Names}} {{.Status}}' | grep device-ws
+# All should show "Up X seconds (healthy)"
+```
+
+Without this, devices fail with: `connection error (host:443): HS: ws upgrade response not 101`
+
+### 5. Vault key — destroyed on restore
+Not recoverable from filesystem. Destroy via UI:
+Settings → Credentials → Vault → Destroy Vault
+
+### 6. Device connection string format
+```
+wss://unms.forefrontwireless.com:443+<token>+allowSelfSignedCertificate
+```