Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,147 @@
+# Admin-AI Migration Plan: Hetzner → app1
+
+**Source:** 178.156.167.181 (Hetzner CPX41, 93% disk)
+**Target:** 152.53.36.131 (app1, RS 4000, 3% disk)
+**DNS:** admin-ai.itpropartner.com
+
+## Current State
+
+### Source (Old AI Box)
+- LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
+- 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
+- Open WebUI (0.9.6) — 121 MB SQLite DB
+- Ollama — 8 local models (~10 GB), 7 cloud-proxy models
+- Qdrant vector DB — small, optional
+- Caddy reverse proxy
+- 226G disk at 93% — critical threshold
+
+### Target (app1)
+- Docker + Compose v5.3.1 already installed
+- Ollama + Open WebUI + n8n already running
+- Caddy 2.6.2 active on 80/443
+- 1TB at 3% used, 32 GB RAM
+- Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com
+
+## Migration Plan
+
+### Phase 0 — Pre-flight (no downtime, no DNS changes)
+
+- [x] Pull LiteLLM image on app1
+- [x] Set up LiteLLM + Postgres docker-compose on app1
+- [x] Export Postgres DB from old → import to app1
+- [x] Add Caddy config for admin-ai.itpropartner.com on app1
+- [ ] Test admin-ai endpoint on app1 via direct IP or internal DNS — **model routing failure, clean restart in progress**
+- [ ] Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b) — not needed, ollama already running on app1
+
+### Phase 1 — Cutover
+
+- [ ] Switch Hermes to fallback chain (OpenRouter deepseek-chat)
+- [ ] Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
+- [ ] Monitor admin-ai endpoint
+- [ ] Run end-to-end model test through app1
+- [ ] Verify vision (gemini-pro-latest) works
+
+### Phase 2 — Restore Primary
+
+- [ ] Switch Hermes back to admin-ai primary
+- [ ] Verify all 22 cron jobs pass
+- [ ] Verify Anita's profile
+
+### Phase 3 — Decommission (after 3 days stable)
+
+- [ ] Keep old box running for rollback
+- [ ] After verification, decom Hetzner AI server
+
+## Detailed Steps
+
+### Step 0: Pull LiteLLM image on app1
+
+```bash
+ssh -i /root/.ssh/itpp-infra root@152.53.36.131
+docker pull ghcr.io/berriai/litellm:v1.84.0
+```
+
+### Step 1: Create LiteLLM docker-compose on app1
+
+Create `/root/docker/litellm/docker-compose.yml` and `/root/docker/litellm/.env`.
+
+The docker compose needs BOTH `.env` (master key + salt key) AND `config.yaml` (database_url + enforce_database). Even in DB mode, LiteLLM requires config.yaml for routing. Without it, models are listed but completions fail silently.
+
+### Step 2: Export/import Postgres DB
+
+On old server:
+```bash
+PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql
+```
+
+SCP to app1 (route through Core if needed):
+```bash
+scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
+scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/
+```
+
+On app1, start Postgres first, then import:
+```bash
+cd /root/docker/litellm && docker compose up -d litellm_postgres
+sleep 5
+PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql
+```
+
+### Step 3: Start LiteLLM on app1 (CRITICAL)
+
+Start LiteLLM only after Postgres import completes AND is healthy:
+
+```bash
+cd /root/docker/litellm && docker compose up -d litellm
+sleep 60  # Prisma migrations take 30-60s on first start with restored DB
+curl -s http://127.0.0.1:4000/health/liveliness  # Should return "I'm alive!"
+```
+
+**⚠️ CRITICAL — model routing failure on first start (Jul 13, 2026):** After DB import, models appear in `/v1/models` but ALL completions fail with "Invalid model name passed." This occurs even with identical SALT_KEY, DB dump, and Docker image. Root cause is a LiteLLM startup race condition — the model routing table takes time to populate from the DB.
+
+**Fix:** Do a clean restart (`docker compose down && docker compose up -d`). Wait for the full startup cycle (30-60s for Prisma migrations + model index rebuild). Test completions, not just the models endpoint. A model in `/v1/models` that fails completions means the routing layer didn't initialize properly on first boot. Verify with:
+
+```bash
+curl -s http://127.0.0.1:4000/v1/chat/completions \
+  -H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Hi"}],"max_tokens":5}'
+# Should return actual completion text, not "Invalid model name"
+```
+
+**⚠️ config.yaml MUST be mounted:** Even in DB mode (`STORE_MODEL_IN_DB=True`), LiteLLM requires `config.yaml` with `general_settings.database_url` and `general_settings.enforce_database: true`. Mount as a volume alongside the compose file.
+
+**⚠️ Healthcheck curl pitfall:** The `ghcr.io/berriai/litellm` image does not include `curl`. A `CMD-SHELL curl` healthcheck produces 6000+ consecutive failures and marks the container `(unhealthy)` permanently. Use `wget -qO-` or a Python-native healthcheck instead.
+
+**⚠️ Separate compose per service:** The old box uses a monolithic `/opt/ai/docker-compose.yml` with 6 containers. The new deployment uses the Docker standard — one service per `/root/docker/<name>/` directory.
+
+### Step 4: Test on app1 via loopback
+
+```bash
+curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA"
+# Should return 121+ models
+```
+
+### Step 5: Add Caddy config on app1
+
+Caddy already configured:
+```
+admin-ai.itpropartner.com {
+    reverse_proxy 127.0.0.1:4000
+}
+```
+
+### Step 6: DNS cutover
+
+Flip DNS A record to 152.53.36.131, verify with curl to admin-ai.itpropartner.com.
+
+## Rollback
+
+If cutover fails:
+1. DNS back to 178.156.167.181
+2. Switch Hermes back to admin-ai (old box still running throughout)
+3. Old box continues operating — zero data loss
+
+## Provider API Keys
+
+All model API keys are encrypted in Postgres using LITELLM_SALT_KEY. As long as the same SALT_KEY is used on app1, the keys migrate transparently via the Postgres dump. Direct provider env vars (OPENAI_API_KEY, etc.) are all empty — everything routes through Postgres-stored keys or OpenRouter.