Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,96 @@
+---
+name: debt-recovery-platform
+description: "Blueprint for the Debt Recovery Experts (DRE) portal — intake, notarization, certified mail, AI claim scoring, legal review, and payment disbursement workflow."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [web]
+tags: [debt-recovery, portal, docusign, notary, certified-mail, stripe-connect]
+---
+
+# Debt Recovery Platform
+
+The DRE platform is a standalone debt recovery portal for contractors/owner-operators to submit unpaid debt claims. The workflow is designed to be zero-touch from the DRE team's side, with full visibility at every step.
+
+## Full Workflow
+
+```
+Contractor signs up & uploads docs
+        ↓
+AI claim analysis (internal score + case comparison)
+        ↓
+Legal review (paralegal/attorney) → "Viable in Texas?"
+        ↓
+LPOA sent to customer → Notarized online via Proof (RON)
+        ↓
+Debtor served → Certified letter via LetterStream
+        ↓
+Debt collected via ACH → Stripe Connect holds funds
+        ↓
+DRE fee deducted → Balance paid to contractor
+        ↓
+Customer tracks everything in portal
+```
+
+## Key Integrations
+
+### Online Notary — Proof (formerly Notarize)
+- **Cost:** $25/notarization (in-house) or $10-25 (on-demand network)
+- **API:** REST with webhooks (Premium tier required)
+- **Texas compliant:** ✅ HB 3496 & SB 1624
+- **Flow:** Customer initiates notary session from within portal → signs LPOA → notarized in same session
+
+### Certified Mail — LetterStream
+- **Cost:** ~$8.34 per certified letter
+- **API:** REST, no monthly minimums
+- **Delivery:** They print, fold, stuff, and mail via USPS
+- **Proof:** Return receipt / certificate of mailing
+
+### Payments — ACH (preferred) via Stripe Connect
+- **Stripe ACH:** 0.8% capped at $5 (vs 2.9% + $0.30 for cards)
+- **Flow:** Debtor pays via ACH → Stripe holds → DRE deducts fee → disburses to customer
+- **Pass-through fees:** Processing costs can be included in what the debtor pays
+
+### Digital Signatures — DocuSeal (self-hosted recommendation)
+- **Why:** Lightweight (227 MB Docker image), SQLite (no PostgreSQL), free embedding SDKs
+- **Covers:** Service agreements, disbursement authorizations, internal approvals
+- **Not for:** LPOA (that requires RON with Proof)
+
+## AI Claim Analysis
+
+A high-end vision+reasoning model (Claude Opus 4.7 or GPT-4o) scores each claim internally:
+
+| Factor | What it evaluates |
+|---|---|
+| Documentation quality | Completeness of contract, invoice, correspondence |
+| Debtor profile | Company type, solvency indicators, responsiveness |
+| Legal standing (TX) | Is the contract valid? Proper jurisdiction? Statute of limitations? |
+| Amount reasonableness | Does the claim amount match the scope of work? |
+
+**Case comparison:** The AI also searches public civil case records for similar disputes — average recovery rate, median resolution time, and how the new claim compares.
+
+The analysis is visible ONLY to the DRE internal team (not the customer).
+
+## Per-Claim Cost Estimate
+
+| Item | Cost |
+|---|---|
+| Notarization (LPOA) | ~$25-35 |
+| Identity verification | ~$4 |
+| Certified mail (debtor) | ~$8.34 |
+| ACH processing | ~$5 max |
+| **Total per claim** | **~$42-52** |
+
+These costs are passed to the debtor as part of the recovery, not borne by DRE.
+
+## Portal Pages
+
+See `/root/portal-mockup/debt-recovery.html` (customer intake) and `/root/portal-mockup/dre-dashboard.html` (internal dashboard with AI analysis).
+
+## Status: Planning Phase
+
+Integrations identified but not yet deployed:
+- Proof (RON) — needs account setup
+- LetterStream — needs account setup  
+- Stripe Connect — needs account setup
+- DocuSeal — can deploy on Core Docker immediately
@@ -0,0 +1,104 @@
+---
+name: infrastructure-audit
+description: "Umbrella for infrastructure audits: S3 backup verification, warm standby failover testing, config file coverage, password audit, systemd service backup, cron health check, and documentation of findings."
+version: 1.0.0
+author: ShoNuff
+tags: [devops, audit, backup, dr, failover, compliance]
+---
+
+# Infrastructure Audit
+
+Use this umbrella for periodic infrastructure health checks and disaster recovery verification. Covers S3 backups, warm standby failover readiness, config file coverage, credential security, cron job health, and system integrity.
+
+## Audit Checklist (run in order)
+
+### 1. S3 Backup Verification
+Check all Wasabi buckets for recency, file counts, and versioning:
+- `hermes-vps-backups` — Hermes config/sessions/profiles (check `live/`, `standby/`, `hermes-full-backup/`)
+- `itpropartner-backups` — portal files, public data
+- `mikrotik-ccr-backups` — router configs
+
+**What to check per bucket:**
+- Exists and is accessible
+- Versioning enabled (MFADelete should be disabled)
+- Most recent upload timestamp — flag anything >48h stale
+- File counts and total size seem reasonable
+- Zero-byte files are expected WAL artifacts only
+
+### 2. Warm Standby Verification
+Check app1-bu (5.161.114.8) via SSH with `itpp-infra` key:
+- Server power state (should be running — warm standby)
+- Hermes gateway status (should be inactive/dormant)
+- AWS CLI installed (`/opt/awscli-venv/bin/activate`)
+- S3 access to `hermes-vps-backups` bucket
+- Cron jobs active: watchdog every 5 min, sync every 10 min
+- Local data freshness (state.db, config.yaml should be recent from sync)
+- Network to Core (ping 152.53.192.33)
+- Disk space (should have >20% free)
+- Latest sync log shows recent activity
+
+### 3. Config & Password Audit
+
+**Password files** should be `chmod 600`:
+| File | Path |
+|------|------|
+| Germaine email | `~/.config/himalaya/g-germainebrown.pass` |
+| Sho'Nuff email | `~/.config/himalaya/shonuff.pass` |
+| iCloud CalDAV | `~/.config/himalaya/g-germainebrown-icloud-calendar.pass` |
+| AWS/Wasabi | `~/.aws/credentials` |
+| SSH keys | `~/.ssh/*` |
+| DRE temp passwords | `~/.hermes/references/dre-temp-passwords.txt` |
+| Migration creds | `~/.hermes/migration-creds.txt` |
+
+**Config files** to check are in backup pipeline:
+- `/root/.hermes/config.yaml` — Hermes config
+- `/etc/caddy/Caddyfile` — Caddy reverse proxy
+- `/root/.hermes/.env` — Environment variables
+- `/etc/systemd/system/*.service` — All custom systemd services
+
+### 4. Cron Job Health
+List all Hermes cron jobs with `cronjob(action='list')` and verify:
+- All critical jobs have `last_status: ok`
+- Full backup job exists (schedule: 0 5 * * *)
+- Live sync job exists (every 15m)
+- Service health check job exists (every 5m)
+- Standby sync job exists on app1-bu (every 10m)
+- Router backups are running
+
+### 5. Documentation
+All findings must be logged in `/root/.hermes/references/dr-issue-log.md`.
+
+**Documentation format preference (Germaine, Jul 8 2026):**
+- **Do NOT use markdown pipe tables in the summary** — they're unreadable on mobile.
+- **Use bullet lists grouped by status** instead. Example:
+  ```
+  **Fixed** ✅
+  - **DR-001** 🔴 Short description → fix summary
+
+  **Resolved** 🟢
+  - **DR-009** Short description
+
+  **Investigating** 🔄
+  - **DR-006** 🔴 Short description
+  ```
+- **Entry structure** — use `Problem → Root Cause → Fix → Verification` blocks with `---` separators between entries.
+
+## Reference files
+| File | Purpose |
+|------|---------|
+| `/root/.hermes/references/dr-issue-log.md` | Permanent issue log, updated on each audit |
+
+## Pitfalls
+- **netcup REST API authentication is complex and poorly documented.** The API uses Keycloak OAuth at `servercontrolpanel.de`, not the CCP API from `customercontrolpanel.de`. There are two separate authentication domains: (1) CCP API key (Master Data → API) for domain/account management, (2) SCP REST API (per-server) for provisioning — requires Keycloak, separate credentials. The CCP key cannot authenticate against the SCP API.
+- **Netcup API requires three credentials:** customer number, API key, and API password (both generated separately from the Master Data page).
+- **When provisioning fails via API,** fall back to: order through the web interface, then I configure via SSH.
+- **Caddyfile is OUTSIDE the Hermes home dir** — it lives at `/etc/caddy/Caddyfile` and is NOT included in `aws s3 sync` of `~/.hermes/`. Must be explicitly added to backup scripts.
+- **Systemd services live in `/etc/systemd/system/`** NOT `~/.config/systemd/user/`. Backup scripts often target the wrong path.
+- **Hetzner API token** may exist as a shell env var but not as `~/.hermes/scripts/.hetzner_token` file (the recovery bundle references the file). Check both sources during audit.
+- **Full backup cron** may not exist — it's a separate cron from the live sync. If missing, create it.
+- **Home-router backup** can fail silently when the MikroTik export produces a `.in_progress` file. Check for stuck export files on the router.
+- **MikroTik CCR backup** requires `paramiko` installed on the backup server. Missing Python deps are a common failure cause.
+- **APP1-bu AWS CLI venv** may be missing on a fresh install. `python3 -m venv` fails without `python3-venv` package. Test with: `source /opt/awscli-venv/bin/activate && aws s3 ls`
+- **Hetzner pricing is inflated for new servers (Jul 2026)** — the API returns current prices which are significantly higher than historical rates. netcup root servers offer better value: RS 1000 (4C/8G/256GB) at €10.74/mo vs Hetzner CPX32 (4C/8G) at ~€35/mo. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
+- **DR issue log formatting:** This user reads on mobile. Pipe tables render poorly. Use grouped bullet lists by status (Fix/Resolved/Investigating), not markdown pipe tables.
+- **Fault is common, not exceptional** — when multiple systems fail in the same audit (stale backups, wrong IPs, missing packages, stuck files), log each as a separate DR issue. Don't treat the cluster of failures as a single incident.
@@ -0,0 +1,70 @@
+---
+name: infrastructure-diagrams
+description: "Build and maintain service dependency diagrams for IT Pro Partner infrastructure — SVG-based architecture maps showing servers, services, cron jobs, email accounts, and dependencies. Generated as self-contained HTML with dark theme."
+version: 1.0.0
+author: Sho'Nuff
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [infrastructure, diagram, dependency, svg, html, architecture]
+---
+
+# Infrastructure Dependency Diagrams
+
+Generate dark-themed SVG dependency diagrams as self-contained HTML files. Design system based on the `architecture-diagram` skill (Cocoon AI style) with modifications for infra-specific layouts.
+
+## When to create or update
+
+- After adding/removing a server, cron job, email account, or external dependency
+- After the user explicitly asks about the dependency map
+- Before making changes to understand service blast radius
+- When user says "keep the diagram updated"
+
+## Diagram structure (7 layers)
+
+1. **External / Users** — Germaine, Telegram Bot
+2. **Messaging / Provider** — Hermes Gateway, admin-ai (LiteLLM)
+3. **Core Services** — Hermes Agent, Cron Scheduler
+4. **Mail Services** — Each email account with its triage/watchdog schedule
+5. **Storage & Backups** — Wasabi S3 buckets
+6. **Standby & External** — Hetzner standby, pending migrations
+7. **Config / Secrets** — config.yaml, .env, Skills/Memory, Vaultwarden
+8. **User VPS** — Tony's CPX21 (Ashburn), standalone Hermes with local Llama fallback
+
+## Color palette
+
+- Hermes Core: cyan (#22d3ee), fill rgba(8,51,68,0.4)
+- System Service: emerald (#34d399), fill rgba(6,78,59,0.4)
+- Email Account: violet (#a78bfa), fill rgba(76,29,149,0.4)
+- External Service: amber (#fbbf24), fill rgba(120,53,15,0.3)
+- Security / Standby: rose (#fb7185), fill rgba(136,19,55,0.3-0.06)
+- External/Generic: slate (#94a3b8), fill rgba(30,41,59,0.5)
+- Cloudflare: amber (#fbbf24) -- already matches the External Service color
+
+## File location
+
+Save to `/root/portal-mockup/dependency-diagram.html`. Serve via Tailscale at the existing `/portal` path alongside other mockups.
+
+## Info cards (3 below diagram)
+
+1. **Hermes Core** — Models used, gateway status, cron job count
+2. **External Dependencies** — admin-ai, Wasabi S3, MXroute, Tailscale
+3. **Single Points of Failure** — What breaks if each external service goes down
+
+## Updating the diagram
+
+After infrastructure changes:
+1. Re-open the existing HTML file
+2. Update SVG element positions/connections
+3. Update info card content
+4. Update the subtitle date
+5. Save with the same filename
+6. The Tailscale Serve path stays valid automatically
+
+## Full dependency reference
+
+The authoritative current-state snapshot lives at `/root/.hermes/references/infrastructure-dependency-snapshot.md`. Update this file whenever infrastructure changes — it provides the source data for regenerating the SVG diagram without re-scanning every service from scratch.
+
+## Keep the diagram updated
+
+This user explicitly said "refer to the your diagram and keep it updated." Treat the diagram as a living document. After any infrastructure change (server added/removed, new cron job, new email account), update the SVG and the dependency snapshot file.
@@ -0,0 +1,46 @@
+# Infrastructure Dependency Snapshot (July 6, 2026)
+
+Live source data for regenerating the SVG dependency diagram.
+
+## Servers
+
+| Name | Provider | Role | Specs | IP | Notes |
+|------|----------|------|-------|-----|-------|
+| netcup | netcup KVM | Live Hermes | 8C/15GB/512GB, Debian 13 | 152.53.192.33 | Primary box |
+| app1 | Hetzner CPX21 | N8N | 3C/4GB | 87.99.144.163 | Needs verification |
+| app1-bu | Hetzner CPX11 | Warm standby | 2C/2GB, Ubuntu 26.04 | 5.161.114.8 | Hermes off, boots from S3 |
+| unms | Hetzner CPX21 | UISP/UNMS | 3C/4GB/80GB | unms.forefrontwireless.com | Pending app3 migration |
+| tony-vps | Hetzner CPX21 | Tony's Hermes | 3C/4GB/80GB, Ubuntu 24.04 | 87.99.159.142 | Ashburn, standalone |
+
+## Home Router (WireGuard)
+
+| Detail | Value |
+|---|---|
+| Model | CCR2004-16G-2S+ |
+| RouterOS | 7.18.2 |
+| Uptime | 14+ weeks |
+| WireGuard IP | 10.77.0.2 |
+| Tunnel latency | ~37ms |
+| SSH | Passwordless via wisp_rsa key through tunnel |
+
+## Cloudflare Integration
+
+| Feature | Token Permission | Status |
+|---|---|---|
+| Zone DNS view/edit | Zone -> DNS -> Edit | Verified (12 zones) |
+| Domain list + expiry dates | Account -> Registrar -> Read | Verified |
+| Domain renew/register | Account -> Registrar -> Admin | Verified |
+| Domain pricing | Account -> Registrar -> Admin | Verified ($10.46/yr .com) |
+
+## Portal Mockups
+
+| Page | Path | Purpose |
+|---|---|---|
+| Login | login.html | Customer login + new customer inquiry |
+| Dashboard | index.html | Admin/customer dual-view |
+| Customer Onboard | onboard.html | Search domain, link services |
+| Router Onboard | onboard-router.html | MikroTik ROS7 script generator |
+| Network Dashboard | network-dashboard.html | All routers, firmware, alerts |
+| Router Detail | router-detail.html | DNS/DHCP/bandwidth drill-down |
+| DNS & Domains | dns.html | Zones, domains, register/transfer |
+| Dependency Diagram | dependency-diagram.html | Full infra map |
@@ -0,0 +1,102 @@
+---
+name: tailscale-infrastructure-access
+description: "Set up Tailscale for private infrastructure access — install on servers, authenticate, standard for services that don't need public exposure."
+version: 1.1.0
+author: ShoNuff
+license: MIT
+platforms: [linux, macos]
+metadata:
+  hermes:
+    tags: [tailscale, vpn, networking, access, security]
+---
+
+# Tailscale Infrastructure Access
+
+Standard for accessing services that don't need public internet exposure. Vaultwarden, future internal tools, and anything without a customer-facing requirement stays behind Tailscale.
+
+## Principles
+
+- **No public ports for internal services** — Tailscale-only, or SSH tunnel as fallback. No DNS records, no Let's Encrypt, no exposed endpoints for administrative tools.
+- **One Tailscale network per infrastructure** — All IT Pro Partner servers join the same tailnet. The user authenticates via their Google account on each device.
+- **Zero-config mesh** — Tailscale handles NAT traversal, DERP relays, and key exchange. No WireGuard configs, no port forwarding, no public IP needed.
+- **UFW must allow the service port** — Tailscale gets traffic to the server, but UFW/iptables still needs to permit the port. Tailscale does not bypass the local firewall. After setting up a Tailscale-only service, verify with `ufw status` and add the port if missing.
+
+## Installation (server side)
+
+```bash
+curl -fsSL https://tailscale.com/install.sh | sh
+tailscale up
+```
+
+This prints a one-time auth URL. Send it to the user to open in their browser (phone, laptop, any device logged into their Google account). The command times out after 15s if the URL isn't opened — that's expected, the URL is still valid. Just re-run `tailscale up` for a fresh URL.
+
+Rename the server from the auto-generated hostname to a clean name:
+```bash
+tailscale set --hostname app1
+```
+
+## Installation (client devices)
+
+| Device | Download |
+|---|---|
+| **macOS** | [tailscale.com/download-mac](https://tailscale.com/download-mac) |
+| **iPhone/iPad** | App Store → "Tailscale" |
+| **Linux** | `curl -fsSL https://tailscale.com/install.sh | sh` |
+
+On first launch, sign in with the same Google account used to authenticate the server. Devices auto-discover each other and appear in `tailscale status`.
+
+## Post-setup
+
+After the server shows `Connected` in `tailscale status`:
+
+```bash
+# Verify connection — lists all connected devices
+tailscale status
+
+# Get the tailnet IP (100.x.x.x)
+tailscale ip -4
+
+# Access services via tailnet IP instead of public IP
+# http://100.x.x.x:8080 → Vaultwarden
+# ssh root@100.x.x.x → SSH via tailnet (if public SSH is closed later)
+```
+
+## Deploying a Tailscale-only service (Vaultwarden pattern)
+
+1. Deploy the service to `~/docker/<service>/` with `docker-compose.yml` + `.env` + `CHANGELOG.md`
+2. Verify it works locally: `curl -s http://localhost:<port>/`
+3. Allow the port through UFW: `ufw allow <port>/tcp comment '<description>'`
+4. Set the service's `DOMAIN` / `PUBLIC_URL` / base URL to the Tailscale access URL — if using raw tailnet IP, set to `http://100.x.x.x:port`. If using Tailscale Serve for HTTPS (recommended — avoids browser mixed-content warnings on Vaultwarden), set it to the `.ts.net` URL:
+   ```bash
+   tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>
+   ```
+   Then set DOMAIN to `https://<hostname>.tail<random>.ts.net`
+5. The service is now available at `https://<hostname>.tail<random>.ts.net/` — valid HTTPS cert, tailnet-only
+6. Once confirmed working, close the UFW port so the service has zero public surface: `ufw delete allow <port>/tcp`
+7. Log the change: `bash ~/.hermes/scripts/changelog.sh "Networking" "Tailscale Serve enabled for <service> at https://..."`
+
+Vaultwarden specific: The web vault is a JavaScript SPA that calls API endpoints from the browser. If DOMAIN is wrong or points to HTTP from HTTPS (or vice versa), the requests fail silently and the spinner never stops. Fix is always docker-compose restart with correct DOMAIN. Additionally, `SIGNUPS_ALLOWED` must be temporarily true for the first user to register, then locked to false. The admin panel at `/admin` uses ADMIN_TOKEN from .env.
+
+## Security consideration
+
+Once Tailscale is operational and all team devices are connected, the public SSH port can be closed (UFW/iptables deny 0.0.0.0/0 :22, allow only tailnet subnet). This eliminates the public attack surface entirely.
+
+Defer this until all team devices have Tailscale installed and confirmed working.
+
+## Service-access policy
+
+| Service | Public | Tailscale-only | Why |
+|---|---|---|---|
+| Vaultwarden | ❌ | ✅ | Passwords — zero public surface |
+| Future internal tools | ❌ | ✅ | Default for non-customer tools |
+| admin-ai (LiteLLM) | ✅ | — | Hermes needs it from its own subnet |
+| Hermes Telegram | ✅ | — | Needs internet for bot polling |
+
+## Pitfalls
+
+- `tailscale up` prints an auth URL and times out after 15s — that's expected. The URL expires after a few minutes. Just re-run `tailscale up` to get a fresh one.
+- The tailscale IP (`100.x.x.x`) can change if the machine re-authenticates. Use the machine name (visible in `tailscale status` or the admin console) for a stable reference.
+- If the server shows "Logged out" after a reboot, run `tailscale up` again.
+- UFW does not auto-deny by default — a new service on a new port is accessible from the public internet unless explicitly blocked. For Tailscale-only services, confirm with `ufw status` that either the port is allowed (at minimum) or the port has a tailnet-only exception (ideal).
+- Vaultwarden's web vault needs the correct DOMAIN — it uses this to build API URLs. If users see a loading spinner that never resolves, check `docker logs vaultwarden` for `POST /identity/connect/token` requests, then fix DOMAIN and restart.
+- To rename a device in Tailscale: `tailscale set --hostname <new-name>`. Takes effect immediately in `tailscale status` and the admin console.
@@ -0,0 +1,95 @@
+---
+name: tailscale-internal-services
+description: "Deploy internal-only services behind Tailscale for zero public exposure — Tailscale Serve for HTTPS, UFW firewall rules, and service configuration patterns."
+version: 1.0.0
+author: ShoNuff
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [tailscale, vpn, networking, security, internal-services, docker]
+---
+
+# Tailscale Internal Services
+
+Standard for deploying services that should not be publicly accessible. All internal services live behind Tailscale — no DNS records, no public ports, no Let's Encrypt.
+
+## Architecture
+
+```
+Device (iPhone/Mac)                     app1 (netcup)
+┌──────────────┐   Tailscale tunnel     ┌─────────────────────┐
+│  Tailscale   │◄──────────────────────►│  tailscale serve     │
+│  connected    │    encrypted mesh       │  ↓                   │
+│              │                        │  Docker container    │
+│  localhost    │                        │  port 8080           │
+└──────────────┘                        └─────────────────────┘
+         │
+         ▼
+  https://<hostname>.tailXXXXX.ts.net
+```
+
+**Key principle:** Tailscale traffic is already encrypted end-to-end. The Tailscale Serve HTTPS layer is only needed for (a) browser security warnings and (b) apps (like Bitwarden) that reject plain HTTP self-hosted URLs. For API-only services, plain HTTP over the tailnet is fine.
+
+## Setup
+
+### 1. Install Tailscale
+```bash
+curl -fsSL https://tailscale.com/install.sh | sh
+tailscale up
+# Auth URL printed — open in browser to log in
+```
+
+### 2. Start a service with Tailscale Serve
+```bash
+# Expose a local Docker port via Tailscale HTTPS
+tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>
+
+# Verify
+tailscale serve status
+# → https://<hostname>.tail<random>.ts.net/ → proxy http://127.0.0.1:<port>
+```
+
+### 3. Configure the service's DOMAIN
+Update the container's `DOMAIN` environment variable to match the Tailscale URL so the service's internal redirects and API URLs work correctly.
+
+### 4. Close the public port
+```bash
+ufw deny <port>/tcp
+ufw reload
+```
+
+### 5. Rename the hostname if the random tailnet name has confusing characters
+```bash
+tailscale set --hostname <service-name>
+# Then re-create serve config (it picks up the new hostname)
+```
+
+## Application-Specific Notes
+
+### Vaultwarden / Bitwarden
+- Use **"Self-hosted environment"** in the Bitwarden app
+- URL: `https://<hostname>.tail<random>.ts.net`
+- If the Tailscale hostname has zeroes or letter-O confusables, rename the node: `tailscale set --hostname vaultwarden`
+- The Bitwarden mobile app may have issues with certain SSL implementations from Tailscale Serve. If the app can't connect, try:
+  - Use the **Custom Environment** option in the app and enter api/identity/web vault URLs separately
+  - Access via pure HTTP on the Tailscale IP (`http://100.x.x.x:8080`) instead of the HTTPS hostname — works fine since tailnet traffic is encrypted
+
+## UFW Rules
+
+```bash
+# Default: ALLOW SSH, HTTP, HTTPS for management
+ufw allow 22/tcp
+ufw allow 80/tcp
+ufw allow 443/tcp
+
+# Per service: start with port open for testing, then close when Tailscale is verified
+ufw allow <port>/tcp comment 'Service name (temp)'
+ufw delete allow <port>/tcp  # After verifying Tailscale access
+```
+
+## Pitfalls
+- **Tailscale Serve requires HTTPS** — you can't serve a port on both HTTP and HTTPS via Tailscale Serve simultaneously
+- **Hostname changes after serve is active** — you need to restart the serve daemon (turn off, then re-enable with the new hostname)
+- **Safari on iOS shows "Not Secure"** for plain HTTP over tailnet — this is cosmetic, the tunnel is still encrypted
+- **`tailscale serve status` shows a tailnet-only badge** — verify with `-o json` for automated checks
+- **Renaming the tailscale node** (e.g. app1→vaultwarden) changes the serve URL immediately without downtime