ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial skills documentation — 25 categories, all SKILL.md + references + scripts

Browse Source
This commit is contained in:
root
2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/server-architecture-plan/SKILL.md
+170
View File
@@ -0,0 +1,170 @@
---
name: server-architecture-plan
description: ITPP infrastructure architecture — server roles, service placement, and the migration plan from Hetzron to netcup. Approved July 2026.
tags: [infrastructure, architecture, netcup, migration, planning, RBAC]
---
# Server Architecture Plan
## Approved Architecture (July 2026) — Rebalanced Jul 12
### Base Server Standard
All new netcup servers are **RS 4000 G12** — 12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe, ~$44/mo. Exception: Core (RS 2000) keeps Hermes + LiteLLM + Ollama fallback. core-bu (Hetzner, was app1-bu) is warm standby, pending CPX11→CPX31 upgrade.
### Server Roles — Post-Rebalance
| Box | Model | Cost | IP | Role |
|-----|-------|------|-----|------|
| **Core** | RS 2000 | ~$24/mo | 152.53.192.33 | **Hermes + MCPs only**. LiteLLM, Caddy, Portal, Super Search, Ollama moved to app1/decommissioned. See audit Jul 13. |
| **app1** | RS 4000 | ~$44/mo | 152.53.36.131 | **Service hub** — Caddy (all domains), Ops Portal, OSINT API, Super Search MCP, SearXNG, Grafana, cron, n8n + Postgres, Docker services |
| **app2** | RS 4000 | ~$44/mo | 152.53.39.202 | **Infrastructure** — UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target) |
| **app3** | RS 4000 | ~$44/mo | 152.53.241.111 | **Web apps** — WordPress/Apex from wphost02/SiteGround |
| **core-bu** (was app1-bu) | CPX11→CPX31 | ~$22/mo | 5.161.114.8 | Warm standby, needs upgrade for DR capacity |
| | | **~$174/mo** | | |
### Hetzner Cleanup Plan (Jul 15, 2026 — mostly complete)
**Deleted today (Jul 15):**
- unifi (178.156.131.57) — ✅ Old UniFi controller decommissioned, backups in S3, server deleted
- hudu (178.156.130.130) — ✅ Full data migration to app2 verified (135 assets, 125 passwords, 18 layouts restored), server deleted
- unms (5.161.225.131) — ✅ UNMS running on app2, full backup to S3 (994MB + 1,107 configs), server deleted
**Previously deleted:**
- fleettracker (178.156.149.32) — ✅ Deleted Jul 14
**Remaining Hetzner servers (4):**
- core-bu (5.161.114.8) — Warm standby, keep
- ai.itpropartner.com (178.156.167.181) — LiteLLM, pending migration to app1
- wphost02 (5.161.62.38) — WordPress, pending migration to app3
- app1.itpropartner.com (87.99.144.163) — Old app1, deprecated
### Why RS 4000 as standard
- 12 dedicated EPYC cores — no shared vCPU contention
- 32 GB ECC RAM — headroom for Java apps (UniFi, Traccar) + 20+ Docker containers
- 1 TB NVMe — Ollama models alone need 50+ GB, plus UNMS/UISP data
- Fault isolation — an outage on app2 doesn't touch Core or app1
## Service Placement
### Core (RS 2000) — Existing, paid
- Hermes Agent (systemd, port 8081)
- Caddy reverse proxy (systemd, ports 80/443)
- Ops Portal backend (FastAPI, systemd, port 8090)
- Prometheus + Grafana (Docker)
- **Uptime-Kuma** (Docker, `/root/docker/uptime-kuma/`, port 3001) — migrated from docker server Jul 13 with 9 monitors + Telegram/Discord/Slack notifications
- FT360 MCP server (FastMCP, systemd, port 8903, `/root/docker/ft360-mcp/`) — 3 tools: get_devices, get_positions, get_stats. Queries Traccar H2 on app2 via SSH+SCP pattern
- Vaultwarden, DocuSeal, SearXNG, Twenty CRM (Docker under /root/docker/*/)
- Tailscale (systemd)
- **Super Search** — FastMCP server at http://127.0.0.1:8899/mcp (systemd: super-search.service). Multi-provider web search: SearXNG → Exa → OpenCorporates → CourtListener → Firecrawl, plus Trafilatura extraction + premium Exa endpoint. All 3 exposed as Hermes MCP tools: `web_search`, `web_extract`, `web_search_premium`. v1.2.0: token-bucket rate limiting, TTL caching, exponential backoff per provider. Deployed at `/root/docker/super-search/server.py` via venv.
### app1 (RS 4000) — 152.53.36.131 — Currently Live
- Ollama (`ollama/ollama:latest`) — running
- Open WebUI (`ghcr.io/open-webui/open-webui:latest`) — running on :3000
- n8n (`n8nio/n8n:latest`) — running on :5678, proxied at n8n.itpropartner.com
- n8n-postgres (`postgres:16-alpine`) — running
- **LiteLLM** — NOT yet deployed (target for admin-ai migration)
- Caddy 2.6.2 — active on 80/443, proxying n8n and Open WebUI
- 1TB disk — 3% used (943 GB free), 32 GB RAM, 12C EPYC
- All deployments per docker-service-deployment standard at /root/docker/*/
See `references/admin-ai-migration-plan.md` for the detailed cutover plan (Phase 0–3).
See `references/core-rebalance-plan.md` (created Jul 12, 2026) for the full service rebalance — moving Caddy, Ops Portal, Super Search, SearXNG, Grafana, cron from Core → app1; LiteLLM from Hetzner old-ai → Core; Ollama fallback from app1 → Core. 6-phase executable plan with DNS cutover and rollback.
### app2 (RS 4000) — New
- UNMS/UISP full stack (10 Docker containers: nginx, API, device-ws, PostgreSQL, SiriDB, RabbitMQ, Fluentd, Netflow)
- UCRM (customer management CRM)
- UniFi Controller (native Java app + MongoDB)
- Hudu (**full migration complete Jul 15** — 21 companies, 135 assets, 125 passwords, 18 custom layouts all verified on app2. Old server 178.156.130.130 deleted. Daily backup cron active: 3 AM ET → S3, 30-day retention)
- Traccar (Docker at `/root/docker/traccar/` — port 8082 web dashboard + 5000-5150 TCP/UDP for GPS device protocols). Cloudflare DNS: fleettracker360.com, gps.fleettracker360.com, www.fleettracker360.com → 152.53.39.202
### app3 (RS 4000) — New
- WordPress/Apex Track Experience (from wphost02 via RunCloud or standalone Nginx)
- Other WordPress sites (from wphost02 / SiteGround)
- Web apps requiring PHP/Nginx stack
### app1-bu (Hetzner CPX11) — Stays
- Warm standby for Core
- Hermes gateway dormant
- StrongSwan + L2TP
- Watchdog checks Core every 5 min, 2 min confirmation window
## Migration Status
### Ready to migrate
- **ai.itpropartner.com** (LiteLLM, Open WebUI, Ollama, Qdrant) → **app1**
- LiteLLM uses PostgreSQL DB for model configs — dump/restore
- Open WebUI data in SQLite at /app/backend/data/webui.db (~121 MB, 1.9 GB total)
- Ollama models at /root/.ollama (need to copy or re-pull)
- CRITICAL: admin-ai.itpropartner.com must have zero downtime. Set up new proxy first, test, switch DNS, decommission old.
- **n8n** (current Hetzner app1) → **app1**
- Docker compose at /opt/n8n/ — migrate volumes
- Benefits from direct Ollama access on same box
### Approved for migration (pending execution)
- UNMS/UISP → app2 (Docker stack ready, DNS pending SiteGround update)
- **UniFi Controller** → app2 (deployed Jul 12 at 152.53.39.202:8443, backup .unf file import needed)
- **Traccar GPS** → app2 (ready for fresh Docker deploy, HERE Maps API for traffic-aware ETA)
- **Hudu → app2** (🔴 PARTIAL — DNS cut over, companies migrated, but assets/passwords tables empty. Old server still up with all data. Full DB re-dump/restore needed. See `docker-service-deployment` skill for verification queries.)
- WordPress → app3
### Stays at Hetzner
- app1-bu (warm standby)
- tony-vps (Anthony's Hermes)
- Container host "docker" (178.156.168.35) — NPM, RustDesk, StrongSwan, RunCloud stack remain. Uptime-Kuma migrated to Core Jul 13. Server decommissioning deferred until RunCloud web apps migrate to app3 and RustDesk/StrongSwan decisions are made.
### API Access Status (Jul 13, 2026)
- **Hetzner API token expired** — server deletion/provisioning requires manual Hetzner Cloud Console access until token is rotated
- **SiteGround DNS** — no API; all DNS changes for itpropartner.com and related domains must be manual through SiteGround web panel
## Estimated Monthly Cost Breakdown
| Server | Provider | Model | Cost |
|--------|----------|-------|------|
| Core | netcup | RS 2000 | ~$24 |
| app1 | netcup | RS 4000 | ~$44 |
| app2 | netcup | RS 4000 | ~$44 |
| app3 | netcup | RS 4000 | ~$44 |
| app1-bu | Hetzner | CPX11 | ~$8 |
| **Total** | | | **~$164/mo** |
| Current Hetzner total | Hetzner | 10 VPS | ~$150/mo (inflated) |
## RBAC Model (for ops portal)
| Role | Scope | Who |
|------|-------|-----|
| **Admin** | Full access — servers, customers, billing, provisioning, DNS, audit logs | Germaine |
| **Tech/Support** | Tickets, server status, network health. No billing, no provisioning. | Future hire |
| **Sales/Onboarding** | Customer activations, provisioning status, order forms | Future hire |
| **Viewer** | Read-only, limited scope | Anita, auditors |
Roles built into ops portal from day one. Not retrofitted.
## Docker Standard Reference
All services at /root/docker/<name>/ with:
- One `docker-compose.yml`, one `.env` (chmod 600)
- Healthchecks on every service
- Resource limits per container
- Named volumes (not bind mounts for app data)
- Internal-only networks for service-to-service traffic
- Logging with rotation (max-size 10m, max-file 3)
- Automated S3 backup of volumes
## Pitfalls
- **admin-ai goes down = Sho'Nuff loses model access.** Fallback chain: deepseek-v4-pro (admin-ai) → deepseek/deepseek-chat (OpenRouter) → cheap API fallbacks. No Ollama. admin-ai lives on app1 for fault isolation — Core and admin-ai should not share the same box.
- **Do NOT put all AI containers in one compose file.** Each service gets its own /root/docker/<name>/ — LiteLLM, Open WebUI are separate.
- **UniFi on app2 needs MongoDB.** Must migrate the native MongoDB database along with the controller. Cannot just copy files.
- **Traccar device ports: 5000-5150 TCP+UDP** — Ensure firewall allows this range. GPS devices connect directly (non-proxied DNS at `gps.fleettracker360.com`). Web dashboard on port 8082.
- **UNMS uses fixed version tags (3.0.140, 4.5.31).** Do not use `:latest` for Ubiquiti images — they break on minor releases.
- **admin-ai placement decided Jul 13: app1, not Core.** LiteLLM migration target is app1 (152.53.36.131). Core keeps Hermes + MCPs only. See architecture audit at `/root/.hermes/references/architecture-audit-2026-07-13.md`.
- **Ollama removed Jul 13.** Too weak for real work, wasted RAM on Core. Replacement: cheap API fallbacks (deepseek-v4-flash, qwen3-flash via OpenRouter). Remove ollama.service from Core, delete llama3.3:70b (42GB disk waste).
- **Caddy + existing containers port conflict (app2)** — When Caddy shares a box with services like UNMS nginx (port 443), Caddy's auto port 80 binding also conflicts. Fix: use `auto_https disable_redirects` + explicit `domain:443` blocks in Caddyfile. Change existing containers to only bind their required ports.
- **Hudu migration: verify ALL tables, not just companies (Jul 15, 2026).** Initial pg_dump/restore only copied the `companies` table — `assets` and `asset_passwords` were empty on the target. Post-migration verification must query every major table (`companies`, `assets`, `asset_passwords`) and compare row counts between old and new before declaring done. Use `curl --resolve` to hit both old and new IPs through the domain for side-by-side API comparison. See `docker-service-deployment` skill for the full pattern.
```
{
default_bind 152.53.192.33
}
```
This tells Caddy to bind only the public IP. Tailscale keeps its Tailnet IP. Both coexist without conflict. Verify with `caddy validate --config /etc/caddy/Caddyfile` then `systemctl start caddy`.
skills/devops/server-architecture-plan/references/admin-ai-migration-plan.md
+147
View File
@@ -0,0 +1,147 @@
# Admin-AI Migration Plan: Hetzner → app1
**Source:** 178.156.167.181 (Hetzner CPX41, 93% disk)
**Target:** 152.53.36.131 (app1, RS 4000, 3% disk)
**DNS:** admin-ai.itpropartner.com
## Current State
### Source (Old AI Box)
- LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
- 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
- Open WebUI (0.9.6) — 121 MB SQLite DB
- Ollama — 8 local models (~10 GB), 7 cloud-proxy models
- Qdrant vector DB — small, optional
- Caddy reverse proxy
- 226G disk at 93% — critical threshold
### Target (app1)
- Docker + Compose v5.3.1 already installed
- Ollama + Open WebUI + n8n already running
- Caddy 2.6.2 active on 80/443
- 1TB at 3% used, 32 GB RAM
- Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com
## Migration Plan
### Phase 0 — Pre-flight (no downtime, no DNS changes)
- [x] Pull LiteLLM image on app1
- [x] Set up LiteLLM + Postgres docker-compose on app1
- [x] Export Postgres DB from old → import to app1
- [x] Add Caddy config for admin-ai.itpropartner.com on app1
- [ ] Test admin-ai endpoint on app1 via direct IP or internal DNS — **model routing failure, clean restart in progress**
- [ ] Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b) — not needed, ollama already running on app1
### Phase 1 — Cutover
- [ ] Switch Hermes to fallback chain (OpenRouter deepseek-chat)
- [ ] Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
- [ ] Monitor admin-ai endpoint
- [ ] Run end-to-end model test through app1
- [ ] Verify vision (gemini-pro-latest) works
### Phase 2 — Restore Primary
- [ ] Switch Hermes back to admin-ai primary
- [ ] Verify all 22 cron jobs pass
- [ ] Verify Anita's profile
### Phase 3 — Decommission (after 3 days stable)
- [ ] Keep old box running for rollback
- [ ] After verification, decom Hetzner AI server
## Detailed Steps
### Step 0: Pull LiteLLM image on app1
```bash
ssh -i /root/.ssh/itpp-infra root@152.53.36.131
docker pull ghcr.io/berriai/litellm:v1.84.0
```
### Step 1: Create LiteLLM docker-compose on app1
Create `/root/docker/litellm/docker-compose.yml` and `/root/docker/litellm/.env`.
The docker compose needs BOTH `.env` (master key + salt key) AND `config.yaml` (database_url + enforce_database). Even in DB mode, LiteLLM requires config.yaml for routing. Without it, models are listed but completions fail silently.
### Step 2: Export/import Postgres DB
On old server:
```bash
PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql
```
SCP to app1 (route through Core if needed):
```bash
scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/
```
On app1, start Postgres first, then import:
```bash
cd /root/docker/litellm && docker compose up -d litellm_postgres
sleep 5
PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql
```
### Step 3: Start LiteLLM on app1 (CRITICAL)
Start LiteLLM only after Postgres import completes AND is healthy:
```bash
cd /root/docker/litellm && docker compose up -d litellm
sleep 60 # Prisma migrations take 30-60s on first start with restored DB
curl -s http://127.0.0.1:4000/health/liveliness # Should return "I'm alive!"
```
**⚠️ CRITICAL — model routing failure on first start (Jul 13, 2026):** After DB import, models appear in `/v1/models` but ALL completions fail with "Invalid model name passed." This occurs even with identical SALT_KEY, DB dump, and Docker image. Root cause is a LiteLLM startup race condition — the model routing table takes time to populate from the DB.
**Fix:** Do a clean restart (`docker compose down && docker compose up -d`). Wait for the full startup cycle (30-60s for Prisma migrations + model index rebuild). Test completions, not just the models endpoint. A model in `/v1/models` that fails completions means the routing layer didn't initialize properly on first boot. Verify with:
```bash
curl -s http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA" \
-H "Content-Type: application/json" \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Hi"}],"max_tokens":5}'
# Should return actual completion text, not "Invalid model name"
```
**⚠️ config.yaml MUST be mounted:** Even in DB mode (`STORE_MODEL_IN_DB=True`), LiteLLM requires `config.yaml` with `general_settings.database_url` and `general_settings.enforce_database: true`. Mount as a volume alongside the compose file.
**⚠️ Healthcheck curl pitfall:** The `ghcr.io/berriai/litellm` image does not include `curl`. A `CMD-SHELL curl` healthcheck produces 6000+ consecutive failures and marks the container `(unhealthy)` permanently. Use `wget -qO-` or a Python-native healthcheck instead.
**⚠️ Separate compose per service:** The old box uses a monolithic `/opt/ai/docker-compose.yml` with 6 containers. The new deployment uses the Docker standard — one service per `/root/docker/<name>/` directory.
### Step 4: Test on app1 via loopback
```bash
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA"
# Should return 121+ models
```
### Step 5: Add Caddy config on app1
Caddy already configured:
```
admin-ai.itpropartner.com {
reverse_proxy 127.0.0.1:4000
}
```
### Step 6: DNS cutover
Flip DNS A record to 152.53.36.131, verify with curl to admin-ai.itpropartner.com.
## Rollback
If cutover fails:
1. DNS back to 178.156.167.181
2. Switch Hermes back to admin-ai (old box still running throughout)
3. Old box continues operating — zero data loss
## Provider API Keys
All model API keys are encrypted in Postgres using LITELLM_SALT_KEY. As long as the same SALT_KEY is used on app1, the keys migrate transparently via the Postgres dump. Direct provider env vars (OPENAI_API_KEY, etc.) are all empty — everything routes through Postgres-stored keys or OpenRouter.
skills/devops/server-architecture-plan/references/consolidation-migration-plan.md
+54
View File
@@ -0,0 +1,54 @@
# Consolidation & Migration Plan
## Strategy
**Target:** Move all Hetzner workloads onto netcup root servers. Hetzner's Jul 2026 pricing makes new Hetzner VPSes uneconomical (3-5x netcup for equivalent compute).
**Keep at Hetzner:** app1-bu (warm standby), tony-vps (Anthony's box)
## Netcup target servers
| Server | Tier | Purpose | Est. Cost/mo |
|--------|------|---------|-------------|
| **Core** (exists) | RS 2000 (8C/16G/512GB) | Hermes, portal, Docker, misc services | ~$20 |
| **Monitor** (new) | RS 1000 (4C/8G/256GB) | AI/LLM, monitoring, automation | ~$12 |
| **Web** (new) | RS 1000 (4C/8G/256GB) | WISP, web apps, docs | ~$12 |
## Migration phases
### Phase 1 — ai.itpropartner.com (CRITICAL: 92% disk)
- Containers: LiteLLM, Open WebUI, Ollama, Qdrant, Caddy
- Move to: Monitor (if exists) or Core temporarily
- Method: docker-compose stack transfer
- Disk issue: 199G of 226G used. Ollama models + Qdrant vectors are the likely culprit
### Phase 2 — unms.forefrontwireless.com + unifi
- unms: 9 containers (UNMS/UISP, UCRM, RabbitMQ, PostgreSQL, Netflow)
- unifi: Native app + MongoDB + MariaDB
- Move to: Web (new RS 1000)
- Method: docker-compose for UNMS, package migration for UniFi
### Phase 3 — hudu.itpropartner.com
- 5 containers: Hudu app, SWAG certbot, PostgreSQL, Redis
- Move to: Web or Monitor (co-locate)
- Method: docker-compose stack transfer
### Phase 4 — n8n, docker host, fleettracker360
- n8n: 2 containers (n8n + PostgreSQL)
- docker: 7+ containers (NPM, RustDesk, Uptime-Kuma, StrongSwan)
- fleettracker360: Traccar (Java, 26+ tracking ports)
- Move to: Core (has capacity: 6% disk, 8 dedicated cores)
- Method: docker-compose stacks + Traccar Java migration
### Phase 5 — wphost02 (review)
- RunCloud + WordPress sites (Apex, BoxPilot, etc.)
- Option A: Move to Web (rsync + DB dump)
- Option B: Stay at Hetzner — keep simple
## Key considerations
- netcup provisioning via SCP REST API: `servercontrolpanel.de/scp-core/api/v1/`
- Auth: Keycloak OIDC password grant (customer# + CCP password)
- New netcup root servers take ~5-15 min to provision
- DNS needs updating at SiteGround after each migration
- Test each service for 24h before decommissioning old box
skills/devops/server-architecture-plan/references/core-rebalance-plan.md
+25
View File
@@ -0,0 +1,25 @@
# Core Rebalance Plan — Service Migration
**Created:** July 12, 2026
**Status:** Planned, not yet executed
## Target Architecture
### Core (152.53.192.33) — Hermes Command Center
Keep: Hermes Agent, LiteLLM, Ollama (lightweight fallback), local scripts, SSH access.
### app1 (152.53.36.131) — Service Hub
Move everything else: Caddy, Ops Portal, OSINT API, Super Search, SearXNG, Grafana, all cron jobs, ops data collector.
### DNS Changes
- ops.itpropartner.com → app1 IP
- core.itpropartner.com → app1 IP
- sign.itpropartner.com → app1 IP
- admin-ai.itpropartner.com → Core IP (stays)
### Phase Plan
1. Prep (30 min) — snapshot, audit app1, stage files
2. Move (2 hr) — services Core → app1
3. LiteLLM (30 min) — Hetzner old-ai → Core
4. Ollama (15 min) — app1 → Core
5. Rename (15 min) — app1-bu → core-bu
skills/devops/server-architecture-plan/references/ip-dns-changes.md
+22
View File
@@ -0,0 +1,22 @@
# IT Pro Partner — Infrastructure Rebalance: IP & DNS Changes
> Last Updated: July 12, 2026
See /root/.hermes/references/ip-dns-changes.md for the full reference document.
## Quick Reference
| Domain | Old IP | New IP | Status |
|--------|--------|--------|--------|
| unifi.itpropartner.com | 143.198.185.17 (DigitalOcean) | 152.53.39.202 (app2) | ✅ DNS updated |
| unms.forefrontwireless.com | 5.161.225.131 (Hetzner) | 152.53.39.202 (app2) | ⏳ Propagation |
| ops.itpropartner.com | 152.53.192.33 (Core) | 152.53.36.131 (app1) | Pending rebalance |
| core.itpropartner.com | 152.53.192.33 (Core) | 152.53.36.131 (app1) | Pending rebalance |
| sign.itpropartner.com | 152.53.192.33 (Core) | 152.53.36.131 (app1) | Pending rebalance |
| admin-ai.itpropartner.com | 178.156.167.181 (old-ai) | 152.53.192.33 (Core) | Pending LiteLLM migration |
## Servers to Cancel
old-ai (CPX41), old app1 (CPX11), docker (CPX11), unms (CPX21), unifi (CPX21), hudu (CPX21), fleettracker (CPX11), wphost02 (CPX21)
Total savings: ~$104/mo
## Servers to Keep
core-bu (CPX11→CPX31), tony-vps (CPX21)
skills/devops/server-architecture-plan/references/master-apps-services.md
+5
View File
@@ -0,0 +1,5 @@
This is a pointer — the canonical file lives at:
/root/.hermes/references/master-apps-services.md
See that path for the complete ITPP Master Apps & Services Inventory.
Created Jul 11, 2026 during a full infrastructure audit across all 8 servers.
skills/devops/server-architecture-plan/references/migration-plan-target-architecture.md
+57
View File
@@ -0,0 +1,57 @@
## Infrastructure Migration Plan (Jul 2026)
### Target Architecture
| Server | Provider | Model | Cost/mo | Role |
|--------|----------|-------|---------|------|
| **Core** | netcup RS 2000 | 8C/16G/512GB | ~$24 (paid) | Hermes, Docker (Vaultwarden, Docuseal, SearXNG, Twenty), Portal, Prometheus, Uptime-Kuma, Caddy, VPN |
| **app2** | netcup RS 4000 | 12C/32G/1TB | ~$44 | UNMS/UISP, UniFi Controller, Hudu, Traccar |
| **app3** | netcup RS 2000 | 8C/16G/512GB | ~$24 | WordPress/Apex, web apps |
| **app1-bu** | Hetzner CPX11 (2C/2G) | Staying | Paid | Warm standby for Core. Keep in place. |
### Migration rules
1. **Zero downtime for admin-ai.itpropartner.com** — LiteLLM (deepseek-chat proxy for Sho'Nuff) must stay live during migration. Set up new LiteLLM on Core first, test via alternate URL, switch DNS.
2. **Admin-ai uses PostgreSQL for model configs** — DB dump/restore, not config file.
3. **Open WebUI conversations are in SQLite** — stop container, copy `/app/backend/data/` (1.9 GB), restore on new box.
4. **Migration order:** AI stack first (most critical), then UNMS/UniFi, then web apps.
5. **app1-bu stays warm at Hetzner** — do not migrate. It's the safety net.
6. **New boxes follow Docker install standard** — `/root/docker/<service>/`, healthchecks, resource limits, internal networks, named volumes, .env chmod 600.
7. **SSH key for new boxes:** Deploy `itpp-infra` key during provisioning.
8. **New boxes provisioned via netcup SCP** (no API for provisioning — use web UI at servercontrolpanel.de).
9. **app2 needs RS 4000** — UNMS (10 containers + Java/PostgreSQL/RabbitMQ) + UniFi (Java + MongoDB + MariaDB) + Hudu (5 containers) + Traccar (Java + 26 ports) needs 12C/32G to avoid contention.
10. **Billing:** Monthly (not annual), Virginia (Manassas) location.
### admin-ai migration — critical path
admin-ai.itpropartner.com (LiteLLM on ai box at Hetzner) serves the deepseek-chat model that powers Sho'Nuff. It is the most critical service. Sequential steps:
1. Set up new LiteLLM + PostgreSQL on Core (or Monitor box)
2. Dump old PostgreSQL: `docker exec litellm_postgres pg_dump -U litellm litellm_db > litellm-dump.sql`
3. Restore on new PostgreSQL
4. Test new proxy: `curl http://new-box:4000/v1/models` — should show all models
5. Update Hermes config to point at new proxy for testing
6. Switch admin-ai DNS record to new IP
7. Keep old ai box running for 7 days as fallback
### Cron after migration
- Core retains: `hermes-live-sync`, `hermes-system-config-sync`, `hermes-docker-sync`, `ops-data-collector`, `shark-scraper`, `home-router-daily-backup` (via WG to 10.77.0.2), `vps-threshold-check`
- app2 gets: local UNMS/UniFi/etc. cron jobs, node_exporter for Prometheus scraping
- app3 gets: WordPress cron, node_exporter
- app1-bu keeps: `hermes-standby-watchdog`, `hermes-standby-sync`
### Cost comparison
| | Current (Hetzner) | Target (netcup) |
|---|---|---|
| Servers | 9 + app1-bu | Core + app2 + app3 + app1-bu |
| Monthly | ~$150-160 | ~$68 (app2 + app3) + Core (paid) |
| Savings | — | **~$82-92/mo** |
### Unchanged services
- Nginx Proxy Manager (stays on docker Hetzner box — remaining services there still need reverse proxy)
- RustDesk (stays on docker Hetzner box)
- StrongSwan Docker (stays on docker Hetzner box)
- RunCloud MariaDB instances on decommissioned Hetzner boxes — not migrated, databases stay with their apps
skills/devops/server-architecture-plan/references/netcup-api-investigation.md
+37
View File
@@ -0,0 +1,37 @@
# netcup API Access Investigation — Jul 8, 2026
## Summary
The netcup REST API was NOT successfully configured. All three API paths returned errors or authentication failures.
## Endpoints Tested
### SCP REST API (servercontrolpanel.de)
- **Endpoint:** `https://www.servercontrolpanel.de/scp-ui/api/v1/servers`
- **Auth tried:** `X-API-Key` header, `Authorization: Bearer` header
- **Result:** HTTP 200 — returns SCP login HTML, not JSON
- **Body contains:** `<nc-unauthorized></nc-unauthorized>` — CCP API key doesn't authenticate SCP Keycloak
- **SCP version:** 2026.0703.095128
### CCP REST API (customercontrolpanel.de)
- **Endpoint:** `/rest/v1/servers`, `/api/v1/servers`
- **Auth tried:** `X-API-Key`, Bearer, query param
- **Result:** HTTP 404 — all paths
### JSON-RPC (ccp.netcup.net)
- **Endpoint:** `https://ccp.netcup.net/run/webservice/servers/endpoint.php?JSON`
- **Auth:** `{"action":"getProducts","param":{"apikey":"..."}}`
- **Result:** 200, but `"Invalid entry for field apikey"` — needs customer# + API password
| Endpoint | HTTP | Auth | Result |
|----------|------|------|--------|
| ccp.netcup.net/webservice/v2_0/ | 404 | SOAP | Not Found |
| ccp.netcup.net/endpoint.php?JSON | 200 | JSON-RPC | "Invalid apikey" |
| customercontrolpanel.de/rest/v1/servers | 404 | X-API-Key | Not Found |
| servercontrolpanel.de/scp-ui/api/v1/servers | 200 | X-API-Key | Returns login page |
| servercontrolpanel.de/scp-ui/api/v1/openapi | 200 | X-API-Key | Same — HTML, not JSON |
## Likely Fix
SCP needs its own API key generated within the SCP interface (not CCP Master Data). Or JSON-RPC with customer number + API password.
## Status
netcup API: UNCONFIGURED. Provisioning via web interface + SSH only.
skills/devops/server-architecture-plan/references/super-search-architecture.md
+59
View File
@@ -0,0 +1,59 @@
# Super Search — MCP Server Architecture Pattern
Updated: 2026-07-12
## Architecture
```
Client (Hermes) → localhost:8899/mcp (FastMCP HTTP)
Tools:
web_search(query, limit=5) — multi-provider chain
web_extract(url) — Trafilatura → Firecrawl
web_search_premium(query) — Exa direct (DRE-grade)
```
## Provider Chain with Rate Limiting
Each provider has a token bucket rate limiter, TTL cache, and the chain falls through on failure:
```
web_search → 1. SearXNG (local, 30 req/s, 2min cache)
→ 2. Exa (paid, 10 req/s, 5min cache)
→ 3. OpenCorporates (free, 1 req/s, 1hr cache) — company records
→ 4. CourtListener (free, ~100/min, 1hr cache) — court cases
→ 5. Firecrawl (paid, 5 req/min, 5min cache)
```
On 429/rate-limit: exponential backoff respecting Retry-After header, then fall to next provider. On 5xx: immediate fallback.
## Files
- `/root/docker/super-search/server.py` — FastMCP server (v1.2.0+)
- `/root/docker/super-search/ratelimit.py` — TokenBucket, TTLCache, retry helpers
- Systemd service: `super-search.service`
- Venv: `/root/docker/super-search/venv`
## Rate Limit Configuration
```python
RATE_LIMITS = {
"searxng": {"tokens": 30, "interval": 1.0},
"exa": {"tokens": 10, "interval": 1.0},
"firecrawl": {"tokens": 5, "interval": 60.0},
"opencorporates": {"tokens": 1, "interval": 1.0},
"courtlistener": {"tokens": 10, "interval": 6.0},
}
CACHE_TTL = {
"searxng": 120, "exa": 300, "firecrawl": 300,
"opencorporates": 3600, "courtlistener": 3600,
}
```
## Adding a New Provider
1. Add rate limit and cache TTL to config dicts in ratelimit.py
2. Add `_newprovider_search()` function decorated with `@rate_limited_search("newprovider")`
3. Add to the `web_search` fallback chain in order
4. Bump version, restart service: `systemctl restart super-search`
5. Verify: `hermes mcp test super-search`
skills/devops/server-architecture-plan/scripts/audit-server.sh
+64
View File
@@ -0,0 +1,64 @@
#!/bin/bash
# audit-server.sh — Comprehensive server inventory script
# Runs remotely via SSH on each Hetzner server.
# Saves output to ~/audit-server-output.txt for collection.
set -euo pipefail
OUTPUT="/tmp/audit-$(hostname)-$(date +%Y%m%d).txt"
exec > "$OUTPUT" 2>&1
echo "=== SYSTEM INFO ==="
echo "Hostname: $(hostname)"
echo "OS: $(cat /etc/os-release 2>/dev/null | head -3)"
echo "Kernel: $(uname -r)"
echo "Uptime: $(uptime -p)"
echo "CPU: $(nproc) cores"
echo "RAM: $(free -h | grep Mem | awk '{print $2}')"
echo "Disk: $(df -h / | tail -1 | awk '{print $3 " used / " $2 " (" $5 ")" }')"
echo ""
echo "=== DOCKER CONTAINERS ==="
if command -v docker &>/dev/null; then
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" || echo "Docker not running"
else
echo "No Docker installed"
fi
echo ""
echo "=== LISTENING PORTS (TCP) ==="
ss -tlnp | grep -v "127.0.0.1\|::1"
echo ""
echo "=== LISTENING PORTS (UDP) ==="
ss -ulnp | grep -v "127.0.0.1\|::1" | head -15
echo ""
echo "=== WEB SERVERS ==="
which nginx apache2 caddy 2>/dev/null || echo "No web servers found"
ls /etc/nginx/sites-enabled/ 2>/dev/null | head -5 || echo "No nginx sites"
echo ""
echo "=== DATABASES ==="
which mysql mysqld psql postgres sqlite3 2>/dev/null || echo "No databases found"
systemctl is-active mariadb mysql postgresql 2>/dev/null || echo "No DB services running"
echo ""
echo "=== SYSTEMD SERVICES (non-standard) ==="
# Custom services (not default Ubuntu/system packages)
systemctl list-units --type=service --state=running --no-pager --no-legend 2>/dev/null | grep -vE "systemd|ssh|cron|rsyslog|networkd|resolved|timesyncd|getty|user@|dbus|polkit|syslog|ufw|apparmor|irqbalance"
echo ""
echo "=== HOME DIRECTORIES ==="
ls /home/ 2>/dev/null || echo "No /home/ users"
echo ""
echo "=== CRON ==="
crontab -l 2>/dev/null || echo "No crontab"
echo ""
echo "=== ROOT SCRIPTS ==="
ls /root/*.sh /root/*.py 2>/dev/null || echo "No root scripts"
echo ""
echo "=== AUDIT COMPLETE ==="
echo "Output file: $OUTPUT"