hermes-skills/skills/devops/infrastructure-automation/references/uisp-vault-and-device-ws-recovery.md

UISP Vault Key & Device-WS Recovery

UISP Vault Key Missing

When restoring a UISP backup to a new server, the Credential Vault key is NOT included in the backup. The key was set interactively during initial install years ago and is not stored anywhere on disk.

Symptoms

• "Vault Key Is Missing" error in UISP web UI
• Cannot access saved device passwords, SNMP strings, SSH credentials

Recovery Options

Option A: Find the original key Check anywhere it might have been documented:

• Password manager (Vaultwarden/Bitwarden, 1Password)
• Hudu documentation
• Initial setup notes/emails
• /home/unms/app/.env on the old server (if it still exists)

Option B: Destroy and recreate (UISP web UI)

1. Log into UISP at https://<uisp-domain>/nms/login
2. Navigate to Settings → Credentials → Vault
3. Use "Destroy Vault" option
4. This deletes ALL stored passwords
5. Creates a new vault with a fresh key
6. Online devices can have new passwords auto-generated

Note: The destroy option is designed for exactly this scenario. UISP pushes new generated passwords to any device that checks in. Offline devices need manual password updates.

Device-WS Containers Not Running

After restoring a UISP backup to a new server, the device-ws Docker containers may be in Created state but not Running. These handle WebSocket connections from devices.

Symptoms

• Device registration fails with: "ws upgrade response not 101"
• Device error log shows connection error: HS: ws upgrade response not 101
• docker ps shows 0 unms-device-ws-* containers running
• docker ps -a shows them as Created

Fix

# Start all device-ws containers
docker compose -f /home/unms/app/docker-compose.yml up -d device-ws
# OR start individually
for i in 1 2 3 4 5 6 7 8 9 10 11; do
  docker start unms-device-ws-$i
done

Wait 10 seconds then verify all are healthy:

docker ps --format '{{.Names}} {{.Status}}' | grep device-ws | grep -v healthy

Root Cause

The UISP docker-compose defines the device-ws service with deploy.replicas: 11 but the containers don't auto-start after a backup restore. They're created but need explicit docker start or a docker compose up -d call.

UISP Backup Locations

Server	Path	Retention
Live UISP (app2)	/home/unms/data/unms-backups/backups/	Auto backups daily at 4 AM
UCRM DB (app2)	/home/unms/data/ucrm/ucrm/data/backup/	Auto backups daily

S3 Backup Cron

Script at /root/.hermes/scripts/unms-backup-sync.sh syncs to:

• s3://hermes-vps-backups/unms-backups/old-unms/ (historical, old server)
• s3://hermes-vps-backups/unms-backups/live/ (current, app2)
• s3://hermes-vps-backups/unms-backups/live/ucrm/ (UCRM DB)

Runs daily at 6 AM ET via cron job unms-backup-sync.

UNMS/UISP TLS Certificates

When unms.forefrontwireless.com DNS points to the new server, obtain certs:

# Stop nginx container to free port 80
docker stop unms-nginx

# Get certs
certbot certonly --standalone --non-interactive --agree-tos \
  -m info@itpropartner.com \
  -d unms.forefrontwireless.com \
  --preferred-challenges http

# Inject into UNMS cert directory
cp /etc/letsencrypt/live/unms.forefrontwireless.com/fullchain.pem /home/unms/data/cert/live.crt
cp /etc/letsencrypt/live/unms.forefrontwireless.com/privkey.pem /home/unms/data/cert/live.key
chown unms:unms /home/unms/data/cert/live.*
chmod 600 /home/unms/data/cert/live.key

# Restart nginx
docker start unms-nginx

Auto-renewal is handled by certbot systemd timer.