# UISP Vault Key & Device-WS Recovery

## UISP Vault Key Missing

When restoring a UISP backup to a new server, the Credential Vault key is NOT included in the backup. The key was set interactively during initial install years ago and is not stored anywhere on disk.

### Symptoms
- "Vault Key Is Missing" error in UISP web UI
- Cannot access saved device passwords, SNMP strings, SSH credentials

### Recovery Options

**Option A: Find the original key**
Check anywhere it might have been documented:
- Password manager (Vaultwarden/Bitwarden, 1Password)
- Hudu documentation
- Initial setup notes/emails
- `/home/unms/app/.env` on the old server (if it still exists)

**Option B: Destroy and recreate (UISP web UI)**
1. Log into UISP at `https://<uisp-domain>/nms/login`
2. Navigate to Settings → Credentials → Vault
3. Use "Destroy Vault" option
4. This deletes ALL stored passwords
5. Creates a new vault with a fresh key
6. Online devices can have new passwords auto-generated

**Note:** The destroy option is designed for exactly this scenario. UISP pushes new generated passwords to any device that checks in. Offline devices need manual password updates.

## Device-WS Containers Not Running

After restoring a UISP backup to a new server, the `device-ws` Docker containers may be in `Created` state but not `Running`. These handle WebSocket connections from devices.

### Symptoms
- Device registration fails with: "ws upgrade response not 101"
- Device error log shows `connection error: HS: ws upgrade response not 101`
- `docker ps` shows 0 `unms-device-ws-*` containers running
- `docker ps -a` shows them as `Created`

### Fix
```bash
# Start all device-ws containers
docker compose -f /home/unms/app/docker-compose.yml up -d device-ws
# OR start individually
for i in 1 2 3 4 5 6 7 8 9 10 11; do
  docker start unms-device-ws-$i
done
```

Wait 10 seconds then verify all are healthy:
```bash
docker ps --format '{{.Names}} {{.Status}}' | grep device-ws | grep -v healthy
```

### Root Cause
The UISP docker-compose defines the device-ws service with `deploy.replicas: 11` but the containers don't auto-start after a backup restore. They're created but need explicit `docker start` or a `docker compose up -d` call.

## UISP Backup Locations

| Server | Path | Retention |
|---|---|---|
| **Live UISP** (app2) | `/home/unms/data/unms-backups/backups/` | Auto backups daily at 4 AM |
| **UCRM DB** (app2) | `/home/unms/data/ucrm/ucrm/data/backup/` | Auto backups daily |

### S3 Backup Cron
Script at `/root/.hermes/scripts/unms-backup-sync.sh` syncs to:
- `s3://hermes-vps-backups/unms-backups/old-unms/` (historical, old server)
- `s3://hermes-vps-backups/unms-backups/live/` (current, app2)
- `s3://hermes-vps-backups/unms-backups/live/ucrm/` (UCRM DB)

Runs daily at 6 AM ET via cron job `unms-backup-sync`.

## UNMS/UISP TLS Certificates

When `unms.forefrontwireless.com` DNS points to the new server, obtain certs:
```bash
# Stop nginx container to free port 80
docker stop unms-nginx

# Get certs
certbot certonly --standalone --non-interactive --agree-tos \
  -m info@itpropartner.com \
  -d unms.forefrontwireless.com \
  --preferred-challenges http

# Inject into UNMS cert directory
cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
chown unms:unms /home/unms/data/cert/live.*
chmod 600 /home/unms/data/cert/live.key

# Restart nginx
docker start unms-nginx
```

Auto-renewal is handled by certbot systemd timer.