Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,50 @@
+# Credential Audit — July 8, 2026
+
+Full audit of every credential stored across the infrastructure. Results:
+
+| Credential | Location | Status |
+|-----------|----------|--------|
+| FIRECRAWL_API_KEY | ~/.hermes/.env | ✅ |
+| CLOUDFLARE_API_TOKEN | ~/.hermes/.env | ✅ |
+| admin-ai API key | config.yaml | ✅ |
+| MYSQL_PASS (Apex DB) | config.yaml | ✅ |
+| PG_DATABASE_PASSWORD (TwentyCRM) | twenty/.env | ✅ |
+| ENCRYPTION_KEY (TwentyCRM) | twenty/.env | ✅ |
+| g@germainebrown.com (IMAP) | himalaya/*.pass | ✅ |
+| shonuff@germainebrown.com (IMAP) | himalaya/*.pass | ✅ |
+| iCloud CalDAV | himalaya/*.pass + config | ✅ Repaired |
+| DRE dre@ (IMAP) | script inline | ✅ |
+| DRE collections@ (IMAP) | script inline | ✅ |
+
+## Bug Found and Fixed: `***` baked into health check script
+
+`service-health-check.sh` had `Authorization: Bearer ***` hardcoded instead of using the `$CLOUD_TOKEN` variable. Hermes secret redactor replaced the actual token with `***` when the script was first written, and that literal string stayed in the file. Every cron run sent `Authorization: Bearer ***` → Cloudflare returned a 401.
+
+**Fix:** Changed to `Authorization: Bearer $CLOUD_TOKEN` and added `${CLOUDFLARE_API_TOKEN:-$(grep ...)}` pattern so both the session env and .env file resolve the credential.
+
+**Lesson:** Scripts that reference `***` in their code were written with the real value at some point, but Hermes redacted it at write time. Always test scripts immediately after creation, especially if they claim to have saved a credential.
+
+Also: CLOUDFLARE_API_TOKEN was only in the session environment, not in `.env`. Cron runs in a clean shell — it couldn't find the token. Added to `.env`.
+
+## iCloud CalDAV Password Repair
+
+The CalDAV password for iCloud calendar (`g-germainebrown-icloud-calendar.pass`) was stale. Apple revokes old app-specific passwords when generating new ones.
+
+**Fix:** Generated new password at appleid.apple.com → App-Specific Passwords → saved to `.pass` file, added CalDAV account config to `~/.config/himalaya/config.toml`, verified via PROPFIND against `https://caldav.icloud.com/` (HTTP 207 = Multi-Status = success).
+
+## Complete Credential Map
+
+All credentials and credential locations for this infrastructure:
+
+| Service | Account | Credential Location | Type | Notes |
+|---------|---------|-------------------|------|-------|
+| Cloudflare | API Token | ~/.hermes/.env (CLOUDFLARE_API_TOKEN) | Bearer token | Zone:DNS:Edit + Registrar:Admin |
+| Firecrawl | API Key | ~/.hermes/.env (FIRECRAWL_API_KEY) | Bearer token | fc- prefix |
+| admin-ai | LLM Provider | ~/.hermes/config.yaml (providers.admin-ai.api_key) | Bearer token | sk- prefix, NOT in .env |
+| germainebrown.com | g@germainebrown.com | ~/.config/himalaya/g-germainebrown.pass | IMAP password | Mail delivered via MXroute |
+| germainebrown.com | shonuff@germainebrown.com | ~/.config/himalaya/shonuff.pass | IMAP password | BCC'd on all sends |
+| iCloud | g@germainebrown.com (CalDAV) | ~/.config/himalaya/g-germainebrown-icloud-calendar.pass | App-specific password | Calendar sync, NOT IMAP |
+| TwentyCRM | PostgreSQL | /root/docker/twenty/.env (PG_DATABASE_PASSWORD) | DB password | Docker compose .env file |
+| TwentyCRM | Encryption key | /root/docker/twenty/.env (ENCRYPTION_KEY) | AES key | For data encryption at rest |
+| Apex DB | MySQL | ~/.hermes/config.yaml (mcp_servers.mysql.env.MYSQL_PASS) | DB password | Tunnel via wphost02 |
+| DRE | dre@/collections@ | Inline in scripts + .env (DRE_EMAIL_PW) | IMAP password | Both use same password |
@@ -0,0 +1,126 @@
+# Admin-AI Authentication Troubleshooting
+
+## Symptom: Master Key from config.yaml returns 401
+
+LiteLLM responds with `401 Authentication Error` even though the master key in `/opt/ai/config.yaml` matches what's in the `.env` file.
+
+### Root Cause
+
+LiteLLM has **two places** where the master key is set:
+
+1. `/opt/ai/config.yaml` — `general_settings.master_key`
+2. `/opt/ai/.env` — `LITELLM_MASTER_KEY` (passed as Docker container env var)
+
+**The env var takes precedence.** The docker-compose.yml passes `LITELLM_MASTER_KEY` from `.env` into the container. When LiteLLM starts, it uses the env var value as the master key and registers a hash of it in the `LiteLLM_VerificationToken` Postgres table.
+
+If `config.yaml` and `.env` have **different** master keys, the config.yaml value is ignored. The API accepts whatever key was set via `LITELLM_MASTER_KEY` at container startup.
+
+### How a Mismatch Happens
+
+- The DB was initialized with one master key value (via `LITELLM_MASTER_KEY` env var at first `docker compose up`)
+- Later, `config.yaml` was edited to a different key value, but `.env` was NOT updated (or vice versa)
+- The config.yaml change has no effect because the env var overrides it
+- The DB still only has the original key's hash registered
+
+### Diagnosis
+
+```bash
+# SSH to the AI server
+ssh -i /root/.ssh/itpp-infra root@178.156.167.181
+
+# 1. Check what the env var actually is inside the running container
+docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LITELLM_MASTER_KEY
+
+# 2. Check what config.yaml says
+grep master_key /opt/ai/config.yaml
+
+# 3. Check what .env says
+grep LITELLM_MASTER_KEY /opt/ai/.env
+
+# 4. Check the DB for registered keys — find the one with real spend (this is the working one)
+PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
+  -c 'SELECT token, key_name, spend FROM "LiteLLM_VerificationToken" ORDER BY spend DESC LIMIT 5;'
+```
+
+### Fix
+
+**Option A — Insert the key hash directly into the DB (recommended when config.yaml/.env match but DB has old key):**
+
+```bash
+# SSH to the AI server
+ssh -i /root/.ssh/itpp-infra root@178.156.167.181
+
+# Compute SHA-256 hash of the master key
+HASH=$(echo -n "sk-your-master-key-here" | sha256sum | cut -d' ' -f1)
+
+# Insert into the LiteLLM_VerificationToken table
+# The token column is the SHA-256 hash, and it's the PRIMARY KEY
+PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
+  -c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
+      VALUES ('$HASH', 'sk-...last4', 'default_user_id', 0, '{}') \
+      ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...last4';"
+```
+
+Verification is instant — no container restart needed. The key is available immediately because LiteLLM checks the DB on every request.
+
+**⚠️ `docker compose restart litellm` does NOT fix this** in LiteLLM v1.84.0 with `store_model_in_db: true`. The master key from config.yaml is NOT re-registered in the DB on restart. You must insert the key hash manually as shown above. This was verified experimentally on Jul 11, 2026.
+
+**Why restart doesn't work:** LiteLLM only seeds the master key into the DB on FIRST startup. After that, the `LiteLLM_VerificationToken` table is authoritative. Changing config.yaml or .env without a corresponding DB insert has no effect. Even restarting the container does not trigger a re-read of the config for key registration — the DB is treated as the source of truth once initialized.
+
+**Option B — Fix the .env to match the config.yaml, nuke the DB, and restart (nuclear option, use only if you can lose existing keys):**
+```bash
+# WARNING: This deletes all existing keys and re-seeds from config
+PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
+  -c "DELETE FROM \"LiteLLM_VerificationToken\";"
+docker compose down litellm && docker compose up -d litellm
+# On first startup with an empty DB, LiteLLM re-registers the config.yaml master key
+```
+
+**Option C — Generate a new virtual key via the working API (if any key works):**
+```bash
+curl -X POST 'https://admin-ai.itpropartner.com/key/generate' \
+  -H "Authorization: Bearer <working-key>" \
+  -H 'Content-Type: application/json' \
+  -d '{"models": ["deepseek-chat","deepseek-v4-pro","gemini-flash-latest","gemini-pro-latest","claude-sonnet-4-6","claude-opus-4-8"],"max_budget": 50}'
+```
+
+### Verification
+
+```bash
+curl -s -o /dev/null -w 'HTTP %{http_code}' \
+  https://admin-ai.itpropartner.com/v1/models \
+  -H "Authorization: Bearer <key>" --connect-timeout 10
+# Should return HTTP 200 (not 401)
+```
+
+## Current State (July 11, 2026)
+
+- admin-ai runs on **178.156.167.181** (Hetzner AI box — NOT app1)
+- `.env` file has `LITELLM_MASTER_KEY=sk-lit...c5b1`
+- `config.yaml` also has `general_settings.master_key: sk-lit...c5b1`
+- These ARE the same value, BUT:
+  - The LiteLLM container was started 6+ days ago under a DIFFERENT `.env` value
+  - The DB's `LiteLLM_VerificationToken` has the OLD key hash registered (spent $232+)
+  - Restarting the container with `docker compose restart litellm` will re-register the current key
+- **Core's config.yaml has NO `admin-ai` provider section** — only `openrouter` and `ollama-local`
+- The working key hash with $232 spend is the one that was set at original container startup
+
+## LiteLLM Config Files
+
+| File | Path | Purpose |
+|------|------|---------|
+| Docker compose | `/opt/ai/docker-compose.yml` | Service definitions, env var injection |
+| Env file | `/opt/ai/.env` | Actual secrets (LITELLM_MASTER_KEY, POSTGRES_PASSWORD, LITELLM_SALT_KEY) |
+| App config | `/opt/ai/config.yaml` | LiteLLM runtime config (minimal in DB mode) |
+| Caddyfile | `/opt/ai/Caddyfile` | TLS termination, reverse proxy to :4000 |
+
+## Table Schema
+
+The `LiteLLM_VerificationToken` table uses `token` (the **hashed** key value) as the primary key. The `key_name` column stores the last 4 chars of the plaintext key for identification (e.g., `sk-...bWPA`).
+
+Key fields:
+- `token` — SHA-256 hash of the actual key (PK)
+- `key_name` — Last 4 chars identifier (e.g., `sk-...bWPA`)
+- `spend` — Cumulative spend in USD
+- `max_budget` — Budget limit (null = unlimited)
+- `blocked` — Boolean, null = not blocked
@@ -0,0 +1,88 @@
+# Admin-AI Cost and Spend Audit
+
+Use this when selecting or validating Hermes' primary model on `admin-ai.itpropartner.com`, especially after surprising spend appears in provider dashboards.
+
+## Key lessons
+
+- Hermes' `admin-ai` provider is a LiteLLM proxy. A model like `gpt-5.5` can bill the OpenAI backend even though Hermes only sees `provider: admin-ai`.
+- Large gateway sessions can make every turn expensive because the full prompt/context may reach 100k-270k+ prompt tokens per call.
+- Always distinguish:
+  - Hermes-facing provider/key: `providers.admin-ai.api_key` in `/root/.hermes/config.yaml`
+  - LiteLLM backend provider/key: OpenAI/Gemini/Anthropic/DeepSeek credentials stored in LiteLLM on the admin-ai server
+  - Runtime fallback chain: top-level `fallback_providers`, verified with `hermes fallback list`
+
+## Current cost-aware model-selection workflow
+
+1. Verify the fallback chain before changing the primary:
+   ```bash
+   hermes fallback list
+   ```
+   Required survival chain:
+   - `deepseek/deepseek-chat` via `openrouter`
+   - `llama3.2:3b` via `ollama-local`
+
+2. Query recent LiteLLM spend by model group on the admin-ai server:
+   ```bash
+   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
+     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
+     \"SELECT COALESCE(model_group,''), COALESCE(custom_llm_provider,''), COUNT(*), ROUND(SUM(spend)::numeric,6), SUM(prompt_tokens), SUM(completion_tokens), SUM(total_tokens), ROUND((SUM(spend)/NULLIF(SUM(total_tokens),0)*1000000)::numeric,6) AS effective_usd_per_mtok, MAX(\\\"startTime\\\") FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= NOW() - INTERVAL '72 hours' AND model_group <> '' GROUP BY model_group, custom_llm_provider HAVING SUM(total_tokens) > 10000 ORDER BY effective_usd_per_mtok ASC NULLS LAST;\""
+   ```
+
+3. For same-day spend matching provider dashboards, group by session/user to find context size issues:
+   ```bash
+   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
+     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
+     \"SELECT COALESCE(session_id,'') as session, COALESCE(\\\"user\\\",'') as \\\"user\\\", model_group, COUNT(*), ROUND(SUM(spend)::numeric,4), SUM(prompt_tokens), SUM(completion_tokens) FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= DATE_TRUNC('day', NOW()) GROUP BY session_id, \\\"user\\\", model_group ORDER BY SUM(spend) DESC NULLS LAST LIMIT 20;\""
+   ```
+
+4. Verify candidate model exists and can answer before switching:
+   ```bash
+   KEY=$(python3 - <<'PY'
+   import yaml
+   c=yaml.safe_load(open('/root/.hermes/config.yaml'))
+   print(c['providers']['admin-ai']['api_key'])
+   PY
+   )
+   curl -sS https://admin-ai.itpropartner.com/v1/chat/completions \
+     -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
+     -d '{"model":"gemini-pro-latest","messages":[{"role":"user","content":"Return exactly OK."}],"max_tokens":120}'
+   ```
+   Do not use too-low `max_tokens` with Gemini; it can spend reasoning tokens and return `content: null`.
+
+5. Make one config change at a time and verify:
+   ```bash
+   cp -a /root/.hermes/config.yaml /root/.hermes/config.yaml.bak.$(date +%Y%m%d-%H%M%S)-pre-model-change
+   hermes config set model.default gemini-pro-latest
+   hermes config set model.provider admin-ai
+   hermes fallback list
+   hermes config show
+   ```
+
+## Cost notes from observed usage
+
+Observed effective costs vary by route/model alias. Prefer measured admin-ai spend over assumptions.
+
+Examples observed in LiteLLM spend logs:
+
+| Model group | Backend | Effective cost signal |
+|---|---|---|
+| `deepseek/deepseek-v4-flash` | DeepSeek | very low; good cheap default candidate |
+| `deepseek-v4-pro` | DeepSeek | low; better than `deepseek-chat` and cost-effective |
+| `gemini-pro-latest` | Gemini | mid-cost; more competent than DeepSeek Chat, far cheaper than GPT/Claude in this setup |
+| `gpt-5.5` | OpenAI | can become expensive fast in large-context Telegram sessions |
+| `claude-sonnet-4-6` | Anthropic | strong but expensive for default use |
+
+## Backend key attribution
+
+When the user sees spend in Google AI Studio or platform.openai.com:
+
+- Query `LiteLLM_SpendLogs` grouped by `model_group`, `custom_llm_provider`, and time range.
+- `api_key` in `LiteLLM_SpendLogs` is the LiteLLM virtual/master key hash used by Hermes, not necessarily the raw upstream provider key.
+- Google/Gemini backend credentials may live encrypted in `LiteLLM_CredentialsTable` and model rows may reference encrypted `litellm_credential_name`; do not claim exact raw key mapping unless you have explicitly decrypted/verified it.
+
+Useful table/column facts:
+
+- Spend table: `"LiteLLM_SpendLogs"`
+- Time column is quoted camelCase: `"startTime"`
+- Model table: `"LiteLLM_ProxyModelTable"`
+- Credential table: `"LiteLLM_CredentialsTable"`
@@ -0,0 +1,58 @@
+# Admin-AI Fallback Plan — 2026-07-14
+
+## Session lesson
+
+During the app1 admin-ai migration, the working model catalog was not enough proof of health. Real `/v1/chat/completions` probes showed which routes were actually usable.
+
+Verified outcomes at the time of this plan:
+
+- `deepseek-v4-pro` via admin-ai: usable; reasoning models need enough `max_tokens` or they may appear blank/length-limited.
+- `deepseek-chat` via admin-ai: usable.
+- `gpt-5.5` via admin-ai: blocked by OpenAI insufficient quota.
+- `claude-opus-4-8` and `claude-sonnet-4-6` via admin-ai: blocked by Anthropic usage limits until 2026-08-01.
+- Gemini worked technically, but Germaine had eliminated Gemini from the normal Sho'Nuff model plan. Do not reintroduce it as a normal fallback just because it is available.
+- Ollama/llama3.x is deprecated for Sho'Nuff production routing. Do not use it as automatic fallback or for LLM cron jobs.
+
+## Target fallback shape
+
+Use explicit tier names so same-router fallback is not confused with provider diversity:
+
+1. **Primary:** `deepseek-v4-pro` via admin-ai.
+2. **Same-router model fallback:** `deepseek-chat` via admin-ai.
+3. **Independent direct provider fallback:** add and test a direct provider such as Qwen/DashScope or Direct DeepSeek.
+4. **Aggregator fallback:** `deepseek/deepseek-chat` via OpenRouter.
+5. **Manual escalation only:** GPT/Claude after quota/usage limits are fixed.
+6. **Manual emergency only:** old AI box / DNS rollback. Ollama is not automatic fallback.
+
+During migration stabilization, it is acceptable to place the independent direct provider before the same-router fallback:
+
+```text
+deepseek-v4-pro/admin-ai -> qwen-direct -> deepseek-chat/admin-ai -> openrouter
+```
+
+Long term, once app1 admin-ai proves stable:
+
+```text
+deepseek-v4-pro/admin-ai -> deepseek-chat/admin-ai -> qwen-direct -> openrouter
+```
+
+## Background job policy
+
+- Prefer `no_agent=true` deterministic scripts for watchdogs and health checks.
+- Any LLM cron job must be pinned explicitly; cron jobs do not inherit conversation fallback.
+- Do not pin cron jobs to Ollama.
+- Do not pin normal jobs to Gemini unless the task is specifically vision or the user re-approves Gemini for that class.
+- Use `deepseek-chat` via admin-ai for light summaries/classification.
+- Use `deepseek-v4-pro` via admin-ai for higher-risk research/reasoning.
+- Use premium GPT/Claude only as explicit escalation when quota/limits are healthy.
+
+## Health-check rule
+
+`/v1/models` is catalog health only. It does not prove encrypted credentials, upstream quota, or completions are working. Production health checks must call `/v1/chat/completions` for every active route and classify failures by route:
+
+- admin-ai primary/fallback failure
+- OpenRouter fallback failure
+- direct-provider fallback failure
+- escalation-only quota failure
+
+GPT/Claude quota failures should be reported as escalation-model unavailable, not as total admin-ai failure.
@@ -0,0 +1,60 @@
+# Admin-AI Model Inventory (Jul 11, 2026)
+
+## Confirmed Models Available
+
+Query: `curl -H "Authorization: Bearer sk-..." https://admin-ai.itpropartner.com/v1/models`
+
+### Direct API Models (keys stored in LiteLLM Postgres)
+
+| Model ID | Provider | Source |
+|----------|----------|--------|
+| deepseek-chat | DeepSeek | Direct API key |
+| deepseek-v4-pro | DeepSeek | Direct API key |
+| deepseek-v4-flash | DeepSeek | Direct API key |
+| deepseek-reasoner | DeepSeek | Direct API key |
+| deepseek/deepseek-coder | DeepSeek | Direct API key |
+| deepseek/deepseek-v4-pro | DeepSeek | Direct API key |
+| deepseek/deepseek-chat | DeepSeek | Direct API key |
+| deepseek/* | DeepSeek | Wildcard route |
+| claude-opus-4-8 | Anthropic | Direct API key |
+| claude-opus-4-7 | Anthropic | Direct API key |
+| claude-sonnet-4-6 | Anthropic | Direct API key |
+| claude-sonnet-4-5 | Anthropic | Direct API key |
+| claude-haiku-4-5 | Anthropic | Direct API key |
+| claude-sonnet-5 | Anthropic | Direct API key |
+| claude-fable-5 | Anthropic | Direct API key |
+| gpt-5.5 | OpenAI | Direct API key |
+| gemini-pro-latest | Google | Direct API key |
+| gemini-flash-latest | Google | Direct API key |
+| gemini/gemini-3.1-pro-preview | Google | Direct API key |
+| gemini/imagen-4.0-* | Google | Direct API key |
+| groq/qwen/qwen3-32b | Groq | Direct API key |
+| groq/openai/gpt-oss-20b | Groq | Direct API key |
+| xai/grok-4.3 | xAI | Direct API key |
+| xai/grok-4-1-fast | xAI | Direct API key |
+| xai/grok-code-fast | xAI | Direct API key |
+| openrouter/* | Various | OpenRouter routing |
+
+### OpenRouter-Routed Models (via admin-ai)
+
+90+ OpenRouter models including:
+- openrouter/deepseek/deepseek-r1, deepseek-chat-v3.1, deepseek-v3.2
+- openrouter/anthropic/claude-opus-4.5/4.6/4.7, claude-sonnet-4/4.5/4.6, claude-3.5-sonnet, claude-3-haiku
+- openrouter/openai/gpt-4o, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-5, gpt-5-chat, gpt-5-codex, gpt-5.1-codex-max, gpt-5.2-codex, gpt-5.2-pro, gpt-5.2-chat, gpt-5-mini, gpt-5-nano, o3-mini, o1
+- openrouter/google/gemini-2.5-pro, gemini-2.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite
+- openrouter/meta-llama/llama-3-70b-instruct
+- openrouter/mistralai/* (6+ models)
+- openrouter/qwen/* (7+ models including qwen3.5 variants)
+- openrouter/x-ai/grok-4
+- openrouter/minimax/* (3 models)
+- openrouter/z-ai/glm-4.6, glm-4.7, glm-5, glm-5.1
+
+## Spend (Cumulative)
+- sk-...bWPA (old master key): $232.83
+- sk-...c5b1 (current master key): $0.29+
+
+## LiteLLM Postgres Tables
+- LiteLLM_ProxyModelTable -- model configs with encrypted API keys
+- LiteLLM_VerificationToken -- registered keys (PK is SHA-256 hash)
+- LiteLLM_SpendLogs -- spend records (column: startTime, camelCase!)
+- LiteLLM_Config -- general config
@@ -0,0 +1,41 @@
+# Admin-AI Model Pricing & Selection Guide
+**Research date:** July 10, 2026
+
+## Current Model Costs (non-OpenRouter models in admin-ai)
+
+| Model | Input $/1M tok | Output $/1M tok | Best For |
+|---|---|---|---|
+| **gemini-flash-latest** | $0.08–0.30 | $0.30–2.50 | Fast, cheap, vision capable — quick Q&A |
+| **deepseek-chat** (V3) | $0.14 | $0.28 | Default generalist — what Sho'Nuff ran on |
+| **deepseek-v4-flash** | ~$0.15 | ~$0.30 | Fast coding tasks |
+| **deepseek/deepseek-coder** | $0.14 | $0.28 | Code-specific |
+| **deepseek-v4-pro** | $0.435 | $0.87 | Strong reasoning, subagent primary — current |
+| **deepseek/deepseek-reasoner** (R1) | $0.55 | $2.19 | Deep reasoning |
+| **claude-haiku-4-5** | $1.00 | $5.00 | Fast Anthropic, vision |
+| **gemini-pro-latest** | $1.25 | $5.00 | Strong vision, multimodal — current vision model |
+| **claude-sonnet-4-6** | $3.00 | $15.00 | Balanced Anthropic |
+| **gpt-5.5** | $5.00 | $30.00 | OpenAI flagship |
+| **claude-opus-4-8** | $5.00 | $25.00 | Best reasoning, vision |
+
+## Cost Multiplier vs deepseek-chat
+
+- deepseek-chat: **1×** baseline
+- deepseek-v4-pro: ~3×
+- gemini-flash: ~1–2×
+- claude-haiku: ~7–18×
+- claude-sonnet: ~21–54×
+- claude-opus: ~36–89×
+- gpt-5.5: ~36–107×
+
+## Recommended Models to Add
+
+| Model | Cost | Provider | Why |
+|---|---|---|---|
+| **Qwen 3.5 27B** | ~$0.35 / $0.35 | Together AI / Groq | Best coding per dollar. SWE-bench 77% |
+| **Mistral Small** | $0.10 / $0.30 | Mistral | Cheapest general-purpose |
+| **Llama 4 70B** | ~$0.60 / $0.90 | Together / Groq | 1M context open-weight generalist |
+| **GPT-5.5 Mini** | $1.50 / $6.00 | OpenAI | Budget GPT |
+| **Gemini 3.1 Flash-Lite** | $0.10 / $0.40 | Google | Cheapest 1M context |
+| **Grok 4 Mini** | $0.30 / $0.50 | xAI | Budget Grok |
+
+The AI server already has a `groq/qwen/qwen3-32b` model — Qwen 3.5 27B would be through Together AI for maximum availability. One Together AI API key (~$5 free credit) unlocks Qwen, Llama 4, and Mistral through a single provider.
@@ -0,0 +1,76 @@
+# Admin-AI Provider Verification
+
+Use this when asked whether admin-ai is configured, whether its credentials work, or whether a model is available.
+
+## 1. Inspect effective config without exposing secrets
+
+Parse YAML instead of grepping isolated lines:
+
+```bash
+python3 - <<'PY'
+import yaml
+p = '/root/.hermes/config.yaml'
+c = yaml.safe_load(open(p)) or {}
+print('model:', c.get('model'))
+for name, cfg in (c.get('providers') or {}).items():
+    print(f'provider={name}')
+    if isinstance(cfg, dict):
+        for k, v in cfg.items():
+            if any(s in k.lower() for s in ('key','token','secret','password')):
+                print(f'  {k}:', 'SET' if v else 'EMPTY')
+            else:
+                print(f'  {k}: {v}')
+print('delegation:', c.get('delegation'))
+PY
+```
+
+Interpretation:
+
+- `providers.admin-ai` with a base URL and non-empty key means configured for the current profile.
+- A model or delegation provider naming admin-ai without a matching usable provider entry is broken configuration.
+- An `auth.json` credential-pool record containing only a fingerprint is not a usable credential and not proof that admin-ai is active.
+
+## 2. Test the configured credential
+
+Extract values in-process so the key is not printed:
+
+```bash
+python3 - <<'PY'
+import json, ssl, urllib.request, urllib.error, yaml
+c = yaml.safe_load(open('/root/.hermes/config.yaml')) or {}
+p = (c.get('providers') or {}).get('admin-ai')
+if not isinstance(p, dict):
+    raise SystemExit('NOT_CONFIGURED: providers.admin-ai is absent')
+base = (p.get('base_url') or p.get('endpoint') or '').rstrip('/')
+key = p.get('api_key')
+if not base or not key:
+    raise SystemExit('INCOMPLETE: base URL or API key missing')
+req = urllib.request.Request(base + '/models', headers={'Authorization': 'Bearer ' + key})
+try:
+    with urllib.request.urlopen(req, timeout=15, context=ssl.create_default_context()) as r:
+        body = json.load(r)
+        ids = sorted(str(x.get('id')) for x in body.get('data', []) if x.get('id'))
+        print('HTTP:', r.status)
+        print('MODEL_COUNT:', len(ids))
+        exact = 'deepseek-chat' in ids
+        print('DEEPSEEK_CHAT_EXACT:', exact)
+        if not exact:
+            print('DEEPSEEK_MATCHES:', [x for x in ids if 'deepseek' in x.lower()])
+except urllib.error.HTTPError as e:
+    print('HTTP:', e.code)
+    print('AUTH_RESULT:', 'REJECTED' if e.code in (401,403) else 'REQUEST_FAILED')
+except Exception as e:
+    print('CONNECTION_FAILED:', type(e).__name__, str(e))
+PY
+```
+
+## 3. Report precisely
+
+Separate these conclusions:
+
+- **Configured:** provider exists in effective config.
+- **Credential valid:** authenticated `/models` returned HTTP 200.
+- **Model visible:** exact ID `deepseek-chat` is present.
+- **Historical metadata only:** admin-ai appears in `auth.json` but is absent from effective config.
+
+Never report credentials as tested if no usable key was available. Never claim `deepseek-chat` exists based on a stale model inventory.
@@ -0,0 +1,64 @@
+# Cron Agent Analysis Constraints
+
+## Context
+
+The `daily-spend-monitor` cron job delivers spend data to the agent via "Script Output" (pre-run script output). The agent then analyzes it. But the agent runs under `approvals.cron_mode: deny`, which heavily restricts what it can do.
+
+## Blocked in Cron Mode
+
+| Capability | Why Blocked |
+|---|---|
+| `execute_code` | Blocked by cron_mode deny — runs arbitrary Python that could subvert shell-string checks |
+| `terminal(pipe=true)` | Tirith blocks curl-to-interpreter patterns (e.g., `curl ... | python3`) |
+| `terminal(background=true)` | `notify_on_complete` blocked by cron_mode; background without notify is useless |
+| `web_extract` on authenticated endpoints | Can't send Bearer token — endpoints return 401 |
+| `curl -H "Authorization: Bearer ..."` | May trigger tirith schemeless-url-to-sink rules |
+| OpenRouter API calls | The stored key returns "User not found" 401 |
+
+## What Works in Cron Mode
+
+| Tool | Notes |
+|---|---|
+| `web_extract` on **public** URLs | Works — no auth needed |
+| `terminal` simple commands | Works if they don't trigger tirith patterns (no pipes, no auth headers, no schemeless URLs) |
+| `read_file`, `search_files` | Full access |
+| `write_file`, `patch` | Full access |
+| `memory`, `skill_manage`, `todo` | Full access |
+| `session_search` | Works — able to recall past session context |
+
+## Pattern: Analyze Pre-Run Script Output Only
+
+The spend pre-run script (`spend-monitor-collect.sh`) uses SSH+PSQL on the admin-ai server, which is the ONLY reliable production path for 24h spend data. The script outputs structured sections:
+
+```
+=== LLM Spend Report — Jul 13 2026 07:00 ===
+--- LiteLLM Key Spend (cumulative) ---
+ key_prefix...suffix | 232.83
+--- Last 24h Total Spend ---
+   153.81
+--- DeepSeek Models Last 24h ---
+ deepseek/deepseek-v4-flash | 0.9886
+--- All Models Last 24h (top 10) ---
+ gpt-5.5                    | 122.6489
+```
+
+The agent should parse this data directly and produce the report without attempting supplementary API calls. The data is already complete enough for the standard report format:
+1. Total spend 24h ✓
+2. Models > $5/day ✓ (from "All Models Last 24h")
+3. OpenRouter balance — state as "Unavailable" ✓ (no need to query)
+4. Keys > $50 cumulative ✓ (from "LiteLLM Key Spend")
+
+## Decision Tree for Daily Spend Cron
+
+```
+Script Output received?
+├── YES → Parse and report from data already collected.
+│         Do NOT attempt supplementary API calls.
+│         If OpenRouter key is mentioned, state "Unavailable" and move on.
+│
+└── NO (empty Script Output) →
+    ├── If admin-ai SSH is down: the script's HTTPS fallback also fails
+    │   because curl-auth is blocked by tirith. Report "Data collection
+    │   failed — both SSH and HTTPS paths blocked by cron security policy."
+    └── Report available partial data or state failure clearly.
+```
@@ -0,0 +1,45 @@
+# Delegation Config Restart Trap — Jul 9, 2026
+
+## The Problem
+Subagents kept failing with HTTP 402 (OpenRouter insufficient credits) even after:
+1. Changing `delegation.model` from `openrouter/anthropic/claude-opus-4.8` to `claude-opus-4-8`
+2. Changing `delegation.provider` to `admin-ai`
+3. Verifying the model name was correct via `curl` directly against admin-ai API
+
+Root cause: The running Hermes gateway process (PID 2757853) cached the old delegation config at startup. The `hermes config set` commands only updated the config file on disk. Subagents spawned from the running gateway still used the old OpenRouter-routed model string.
+
+## The Fix
+```bash
+# Stop the running gateway first — must be done from OUTSIDE the gateway session
+# (e.g. via SSH, or from a separate terminal, or from host CLI session)
+hermes gateway stop
+# Or kill the PID directly:
+kill 2757853
+# Then restart:
+systemctl start hermes
+# Or:
+hermes gateway run --replace
+```
+
+**Cannot restart from inside the gateway:** `hermes gateway restart` from inside the Telegram session kills the process before the command completes (SIGTERM propagates to child processes). The error message: "Blocked: cannot restart or stop the gateway from inside the gateway process."
+
+## Symptoms
+- `hermes config get delegation.model` shows the NEW value (config file updated)
+- But subagents still use the OLD model name
+- Subagents fail with `HTTP 402: openrouter...Insufficient credits`
+- The running gateway process (check via `ps aux | grep hermes.*gateway`) has a start time before the config change
+
+## Prevention
+- After ANY `hermes config set` for `delegation.*`, note whether the gateway needs a restart
+- Verification: check the gateway PID's start time vs the config file's modification time
+- If config modified after gateway start → restart needed
+- For known-good behavior: use `deepseek-chat` (the conversation model) as delegation model too — it doesn't go through OpenRouter and doesn't need config changes
+
+## Effective delegation state (Jul 9, 2026, 10:30 PM)
+Despite config file having `model: deepseek-chat`, the running gateway still used the cached old config. All subagents dispatched tonight actually ran on `deepseek-chat` because the config was changed FROM that value first (was deepseek-chat → changed to openrouter/opus failed → changed back). The initial successful deepseek-chat dispatches worked before the first config change attempt.
+
+## Re-confirmed Jul 10, 2026
+
+Same pattern recurred when changing delegation model from `gemini-pro-latest` to `deepseek-v4-pro`. Config confirmed correct, but subagents kept dispatching on Gemini until gateway was restarted. This trap is persistent — it will hit every model change regardless of how carefully the config is verified.
+
+**Cron job drift is a related symptom:** When the conversation model changed from `deepseek-chat` to `deepseek-v4-pro`, unpinned cron jobs refused to run with a safety warning: "global inference config drifted since this job was created." Fix: pin each job explicitly with `hermes cron update <id> --model <model> --provider <provider>`.
@@ -0,0 +1,28 @@
+# Gemini API Quotas & Rate Limits (Jul 10, 2026)
+
+## Google Gemini API Key — 250 Requests/Day Hard Limit
+
+The Gemini API key used through admin-ai (LiteLLM) has a **hard daily limit of 250 requests**. This applies to:
+- `gemini-pro-latest` (used for subagents, vision, and some direct calls)
+- `gemini-3.1-pro` (same quota pool)
+- Vision/image analysis requests count against the same quota
+
+**Error pattern:** `429 RESOURCE_EXHAUSTED — Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model_per_day, limit: 250`
+
+**When it happens:** After ~250 total requests across all consumers (subagents, vision, direct API calls). Typically exhausts mid-afternoon on heavy usage days.
+
+**Recovery:** Quota resets at midnight UTC. Retry delay from error: ~7.5 hours.
+
+## Mitigations
+
+1. **Subagents use deepseek-v4-pro** — avoids Gemini's 250/day cap entirely
+2. **Cron jobs use Ollama** (llama3.2:3b) for low-end tasks — free, no quota
+3. **Vision falls back to text-only** when Gemini is exhausted — describe what you need instead
+4. **Monitor usage** via admin-ai's spend dashboard
+
+## Quota vs Credits
+
+This is a **request count quota**, NOT a credit balance issue. Topping up credits won't fix it. The fix is:
+- Upgrade Google API plan for higher daily limits
+- Or route vision through a different provider (Anthropic's Claude vision, OpenAI's GPT-4o vision)
+- Or accept the 250/day cap and manage usage accordingly
@@ -0,0 +1,84 @@
+# LiteLLM admin-ai migration: encrypted DB rows require matching keys
+
+## When this applies
+
+Use this during `admin-ai.itpropartner.com` / LiteLLM host migrations when:
+
+- `/v1/models` returns HTTP 200 and lists models, but `/v1/chat/completions` hangs, times out, or returns model/auth/provider errors.
+- LiteLLM logs show `CryptoError: Decryption failed` or `Error decrypting value for key: api_key` / `organization`.
+- A PostgreSQL DB dump/restore was moved to a new host.
+
+The model catalog can look healthy even while completions are broken because the restored DB rows contain encrypted provider credentials/model parameters. The new host must use the same `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` that encrypted those rows.
+
+## Safe diagnosis: compare hashes only
+
+Do **not** print secret values. Compare fingerprints between old and new hosts:
+
+```bash
+for host in 'old 178.156.167.181 /opt/ai/.env' 'new 152.53.36.131 /root/docker/litellm/.env'; do
+  label=$(echo "$host" | awk '{print $1}')
+  ip=$(echo "$host" | awk '{print $2}')
+  envfile=$(echo "$host" | awk '{print $3}')
+  echo "===== $label $ip ====="
+  ssh -i /root/.ssh/itpp-infra root@$ip "python3 - <<'PY'
+from pathlib import Path
+import hashlib, os
+p = Path('$envfile')
+vals = {}
+for line in p.read_text().splitlines():
+    if '=' in line and not line.lstrip().startswith('#'):
+        k,v = line.split('=',1)
+        if k in ('LITELLM_MASTER_KEY','LITELLM_SALT_KEY','DATABASE_URL','STORE_MODEL_IN_DB'):
+            vals[k]=v
+for k in ('DATABASE_URL','LITELLM_MASTER_KEY','LITELLM_SALT_KEY','STORE_MODEL_IN_DB'):
+    v = vals.get(k)
+    if not v:
+        print(f'{k}=<missing>')
+    elif 'KEY' in k or 'URL' in k:
+        print(f'{k}=sha256:{hashlib.sha256(v.encode()).hexdigest()[:16]} len:{len(v)}')
+    else:
+        print(f'{k}={v}')
+PY"
+done
+```
+
+Expected: `DATABASE_URL`, `LITELLM_MASTER_KEY`, and `LITELLM_SALT_KEY` fingerprints should match between the host that produced the DB and the host running the restored DB. If master/salt differ, restored encrypted credentials cannot decrypt.
+
+## Minimal fix when old host is still available
+
+Scope: app1/new host only. Do not touch Core Hermes config during this repair.
+
+1. Back up new host `.env`.
+2. Copy only `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` from the old host's LiteLLM `.env` to the new host's LiteLLM `.env`.
+3. Restart only the new host LiteLLM container.
+4. Verify fingerprints match.
+5. Verify both catalog and completions.
+
+```bash
+# On the new host after updating .env:
+cd /root/docker/litellm
+cp .env ".env.before-keyfix-$(date +%Y%m%d%H%M%S)"
+docker compose restart litellm
+```
+
+## Required verification
+
+Do not declare success after `/v1/models` only. Test real completions:
+
+```bash
+KEY='<admin-ai virtual key from Hermes config>'
+for model in gpt-5.5 deepseek-v4-pro deepseek-chat gemini-pro-latest; do
+  echo "=== $model ==="
+  curl -sS --max-time 45 http://127.0.0.1:4000/v1/chat/completions \
+    -H "Authorization: Bearer $KEY" \
+    -H 'Content-Type: application/json' \
+    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply OK only.\"}],\"max_tokens\":16}" \
+    | python3 -m json.tool | head -40
+done
+```
+
+Success means each returns `choices[0].message.content`, not just HTTP 200.
+
+## If old keys are unavailable
+
+Do not brute force or hand-edit encrypted DB rows. Recreate LiteLLM credentials/models fresh on the new host via LiteLLM API/UI or a known-good model import process. Keep the old AI server running until the new host passes completion tests for the core models and at least one full day of clean usage.
@@ -0,0 +1,63 @@
+# LiteLLM HTTPS Spend Query Endpoints (Fallback for SSH)
+
+When SSH to the admin-ai server is unavailable (key not on this box, server temporarily unreachable), spend data can be queried directly from the LiteLLM proxy's HTTPS REST API at `https://admin-ai.itpropartner.com`.
+
+## Auth
+
+Extract the admin-ai API key from Hermes config:
+
+```bash
+KEY=$(grep -A2 'admin-ai:' ~/.hermes/config.yaml | grep api_key | awk '{print $2}')
+```
+
+## Endpoints
+
+### `/global/spend` — Total cumulative spend across all keys
+
+```bash
+curl -s https://admin-ai.itpropartner.com/global/spend \
+  -H "Authorization: Bearer $KEY"
+```
+
+Returns: `{"spend": 228.20, "max_budget": 0.0}`
+
+`max_budget: 0.0` means unlimited (no budget cap set).
+
+### `/spend/keys` — Per-key cumulative spend + metadata
+
+```bash
+curl -s https://admin-ai.itpropartner.com/spend/keys \
+  -H "Authorization: Bearer $KEY"
+```
+
+Returns JSON array where each entry has:
+- `key_name` — last-4-char suffix (e.g. `sk-...bWPA`)
+- `key_alias` — human-readable alias (e.g. `"Hermes"`, `"agent.iamgmb.com"`)
+- `spend` — cumulative dollar amount
+- `last_active` — ISO timestamp of last usage
+- `models` — list of models this key has access to
+- `max_budget` — budget cap (null = none)
+- `created_at` / `updated_at` — timestamps
+
+### `/spend/tags` — Spend broken down by credential/tag
+
+```bash
+curl -s https://admin-ai.itpropartner.com/spend/tags \
+  -H "Authorization: Bearer $KEY"
+```
+
+Returns JSON array where each entry has:
+- `individual_request_tag` — e.g. `"Credential: Deepseek"`, `"User-Agent: OpenAI/Python 2.24.0"`
+- `total_spend` — cumulative dollars
+- `log_count` — number of requests
+
+## Limitations
+
+- These are **cumulative lifetime** figures. For 24h-only data, use the SSH+Postgres query (the `LiteLLM_SpendLogs` table has timestamps).
+- The HTTPS API does not expose per-model spend breakdown for the last 24h — only cumulative per-key and per-tag.
+- The OpenRouter API key in config.yaml typically returns 401 "User not found" when queried — the stored key is dead/expired. OpenRouter credit balance cannot be queried from this box. State as "Unavailable" in reports. Check manually at https://openrouter.ai/settings/credits.
+- **Cron agent analysis cannot use these HTTPS endpoints.** The `cron_mode: deny` setting blocks curl-pipe patterns (tirith blocks curl | python3) and `web_extract` can't send auth headers. These endpoints are useful only for interactive sessions or the pre-run collection script.
+
+## Data Cross-Check
+
+The `/spend/keys` cumulative per-key figures match the Postgres `LiteLLM_VerificationToken` table. The `/global/spend` total may differ slightly from the sum of per-key spends because `/global/spend` is the proxy's internal counter (slightly behind real-time), while `/spend/keys` reads from the DB directly.
@@ -0,0 +1,51 @@
+# LiteLLM NULL-Model Failure Entries in SpendLogs
+
+## Pattern
+
+Rows in `LiteLLM_SpendLogs` where `model`, `model_group`, AND `custom_llm_provider` are all empty/NULL, status is `failure`, spend is 0. These are requests that **failed before model routing** — LiteLLM's `user_api_key_auth()` raised an exception before it could identify a model.
+
+## Real Example (Jul 14, 2026)
+
+```
+ model | model_group | custom_llm_provider | status  | count
+-------+-------------+---------------------+---------+-------
+       |             |                     | failure |   637
+```
+
+637 out of 1,305 requests (49%) were unlabeled auth failures.
+
+## Source
+
+The LiteLLM log shows `auth_exception_handler.py:95` firing repeatedly:
+
+```
+LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -
+```
+
+These are almost certainly **NOT Hermes traffic**. Likely sources:
+- **OpenWebUI** on the same app1 server, hitting the proxy with stale/invalid credentials
+- External services or old integrations with expired API keys
+- Automated health checks using dead keys
+
+## In Spend Reports
+
+When reporting failure rates, **exclude NULL-model entries** from the "Hermes failure rate" calculation. They distort the picture. Report them separately as "unlabeled auth failures (external)" and note they're likely not Hermes traffic.
+
+To filter them out:
+
+```sql
+SELECT ...
+FROM "LiteLLM_SpendLogs"
+WHERE "startTime" >= '...' AND "startTime" < '...'
+  AND (model != '' AND model IS NOT NULL)
+```
+
+## Verification
+
+To confirm they're auth failures (not routing/model issues), check the LiteLLM container logs:
+
+```bash
+ssh app1 "docker logs litellm --since 24h 2>&1 | grep -c 'auth_exception_handler'"
+```
+
+Consistent auth-exception counts matching the NULL-model failure count confirm the pattern.
@@ -0,0 +1,49 @@
+# LiteLLM Spend & Cost Analysis Queries
+
+When investigating high model costs or checking the effective price of models routed through `admin-ai`, you can query the `LiteLLM_SpendLogs` table directly on the AI server (178.156.167.181). 
+
+*Note: Column names in this table use camelCase (e.g., `"startTime"`), which require double quotes in Postgres.*
+
+## 1. Effective Cost per 1 Million Tokens (Last 72 Hours)
+
+This query groups by model and provider, calculates the actual effective cost per 1M tokens based on recorded spend, and filters out trivial usage (<10k tokens). This is the best way to compare real-world pricing for models like `deepseek-v4-pro` vs `deepseek-chat` to make cost-effective primary model recommendations.
+
+```bash
+ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
+  "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \"
+SELECT 
+  COALESCE(model_group,'') as model, 
+  COALESCE(custom_llm_provider,'') as provider, 
+  COUNT(*) as calls, 
+  ROUND(SUM(spend)::numeric,6) as total_spend_usd, 
+  SUM(total_tokens) as tokens, 
+  ROUND((SUM(spend)/NULLIF(SUM(total_tokens),0)*1000000)::numeric,6) AS effective_usd_per_1M_tokens
+FROM \\\"LiteLLM_SpendLogs\\\" 
+WHERE \\\"startTime\\\" >= NOW() - INTERVAL '72 hours' AND model_group <> '' 
+GROUP BY model_group, custom_llm_provider 
+HAVING SUM(total_tokens) > 10000 
+ORDER BY effective_usd_per_1M_tokens ASC NULLS LAST;\""
+```
+
+## 2. Daily Spend by User & Session (Context Size Investigation)
+
+High API costs are often driven by enormous prompt contexts sent repeatedly. This query finds the exact sessions and models burning the most spend today.
+
+```bash
+ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
+  "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \"
+SELECT 
+  COALESCE(session_id,'') as session, 
+  COALESCE(\\\"user\\\",'') as \\\"user\\\", 
+  model_group,
+  COUNT(*) as calls, 
+  ROUND(SUM(spend)::numeric,4) as spend, 
+  SUM(prompt_tokens) as prompt_tokens, 
+  SUM(completion_tokens) as completion_tokens
+FROM \\\"LiteLLM_SpendLogs\\\" 
+WHERE \\\"startTime\\\" >= DATE_TRUNC('day', NOW())
+GROUP BY session_id, \\\"user\\\", model_group 
+ORDER BY SUM(spend) DESC NULLS LAST LIMIT 20;\""
+```
+
+If you see a session averaging 250,000+ `prompt_tokens` per call, the agent's conversation history has grown too large for expensive models like `gpt-5.5` or `claude-sonnet-4-6`. The immediate fix is to start a `/new` session to drop context, or swap the primary model to a cheaper alternative.
@@ -0,0 +1,104 @@
+# LiteLLM Spend Query Patterns (app1 Postgres)
+
+Quick-reference for querying today's spend directly from LiteLLM's Postgres database on app1.
+
+## SSH Access
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131
+```
+
+## Today's Total Spend
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  ROUND(SUM(spend)::numeric, 4) as total_spend,
+  COUNT(*) as total_requests,
+  SUM(total_tokens) as total_tokens
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\'';
+"'
+```
+
+## Per-Model Breakdown
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  model,
+  COUNT(*) as requests,
+  ROUND(SUM(spend)::numeric, 4) as spend,
+  SUM(total_tokens) as tokens
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY model
+ORDER BY spend DESC;
+"'
+```
+
+## Per-User/Key Breakdown
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  COALESCE(u.user_email, s.user, s.api_key) as identity,
+  COUNT(*) as requests,
+  ROUND(SUM(s.spend)::numeric, 4) as spend,
+  SUM(s.total_tokens) as tokens
+FROM \"LiteLLM_SpendLogs\" s
+LEFT JOIN \"LiteLLM_UserTable\" u ON s.user = u.user_id
+WHERE s.\"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND s.\"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY identity
+ORDER BY spend DESC;
+"'
+```
+
+## Failed vs Successful Requests
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT status, COUNT(*), ROUND(SUM(spend)::numeric, 4) as total_spend
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY status
+ORDER BY count DESC;
+"'
+```
+
+## Pitfalls
+
+- **Column names are camelCase**: `startTime`, `endTime`, `completionStartTime` — always quoted
+- **Table name is `LiteLLM_SpendLogs`** (capital L in Logs)
+- **Database is `litellm_db`**, not `litellm`
+- **Single-quote escaping** in nested SSH+Docker+psql is tricky: use `'\\''` to break out of the outer single quote, insert a literal `'`, then resume
+- **Failed requests have `spend = 0` and `total_tokens = 0`** — filter with `WHERE spend > 0` for spend-only analysis
+- **The viewer key (`sk-dZ6QafZLLN5Ewl0NxVlRhQ`) only has `llm_api_routes`** — cannot query `/global/spend` or `/spend/logs` via HTTPS. Go through Postgres directly.
+- **NULL-model failure rows are external auth noise.** Rows with empty `model`, `model_group`, and `custom_llm_provider` + `status = 'failure'` are requests rejected before model routing (auth failures from OpenWebUI or stale integrations). Filter them out for Hermes-specific analysis: `WHERE (model != '' AND model IS NOT NULL)`. See `references/litellm-null-model-failures.md`. These rows inflate failure counts in reports — report them separately.
+
+## Column Reference
+
+Key columns in `LiteLLM_SpendLogs`:
+
+| Column | Type | Notes |
+|---|---|---|
+| `startTime` | timestamp | Request time (server local) |
+| `spend` | double precision | Cost in USD |
+| `total_tokens` | integer | Total tokens consumed |
+| `prompt_tokens` | integer | Input tokens |
+| `completion_tokens` | integer | Output tokens |
+| `model` | text | Model name as called |
+| `model_group` | text | Model group (e.g., `deepseek-v4-pro`) |
+| `custom_llm_provider` | text | Provider (e.g., `deepseek`, `openrouter`) |
+| `user` | text | User ID |
+| `api_key` | text | API key hash used |
+| `status` | text | `success` or `failure` |
+| `cache_hit` | text | `true` if cache hit |
@@ -0,0 +1,65 @@
+# Model Configuration History
+
+Tracks the effective state of all model chains over time. This file is the authoritative reference when debugging fallback behavior or config staleness.
+
+## Current Effective State (Jul 10, 2026)
+
+All chains configured on disk via `hermes config set`. Gateway has NOT been restarted — these apply on next start.
+
+### Sho'Nuff Conversation Chain
+
+| Level | Model | Provider | Cost In/Out per 1M |
+|-------|-------|----------|---------------------|
+| Primary | deepseek-v4-pro | admin-ai | $0.435 / $0.87 |
+| Fallback 1 | deepseek-chat | admin-ai | $0.14 / $0.28 |
+| Fallback 2 | gemini-flash-latest | admin-ai | $0.08–0.30 / $0.30–2.50 |
+| Fallback 3 | llama3.2:3b | ollama-local (Core) | Free |
+
+### Delegation (Subagent) Chain
+
+| Level | Model | Provider | Cost In/Out per 1M |
+|-------|-------|----------|---------------------|
+| Primary | deepseek-v4-pro | admin-ai | $0.435 / $0.87 |
+| Fallback 1 | deepseek-chat | admin-ai | $0.14 / $0.28 |
+| Fallback 2 | gemini-flash-latest | admin-ai | $0.08–0.30 / $0.30–2.50 |
+| Fallback 3 | llama3.2:3b | ollama-local (Core) | Free |
+
+### Anita's Chain
+
+| Level | Model | Provider |
+|-------|-------|----------|
+| Primary | deepseek-v4-pro | admin-ai |
+| Fallback | llama3.2:3b | ollama-local |
+| Vision | gemini-pro-latest | admin-ai |
+
+### Vision
+
+Both profiles (default + anita): `gemini-pro-latest` via admin-ai. Direct routing (NOT OpenRouter).
+
+### API Key
+
+Master key `sk-lbh...bWPA` in all config paths. Virtual keys were tested but reverted on user request.
+
+## Pricing Comparison
+
+| Model | Input $/1M tok | Output $/1M tok | Strengths |
+|-------|---------------|----------------|-----------|
+| gemini-flash-latest | $0.08–0.30 | $0.30–2.50 | Fast, cheap, vision |
+| deepseek-chat (V3) | $0.14 | $0.28 | Good generalist, 1M context |
+| deepseek-v4-flash | ~$0.15 | ~$0.30 | Fast inference, coding |
+| deepseek-v4-pro | $0.435 | $0.87 | Strong reasoning, 1M context |
+| claude-haiku-4-5 | $1.00 | $5.00 | Fast, vision capable |
+| gemini-pro-latest | $1.25 | $5.00 | Strong vision, multimodal |
+| claude-sonnet-4-6 | $3.00 | $15.00 | Balanced, reliable |
+| gpt-5.5 | $5.00 | $30.00 | OpenAI flagship |
+| claude-opus-4-8 | $5.00 | $25.00 | Best reasoning, vision |
+
+## Recommended Use By Cost
+
+| Use Case | Model | Why |
+|----------|-------|-----|
+| Main model | deepseek-v4-pro | $0.435/$0.87 — handles 99% of work |
+| Vision / image analysis | gemini-pro-latest | Direct routing, no OpenRouter credits |
+| Heavy subagent tasks | deepseek-v4-pro | 6× cheaper than Opus, strong reasoning |
+| Only if needed | claude-opus-4-8 | Best model but 89× more expensive |
+| Fast cheap coding | deepseek-v4-flash | Available on admin-ai |
@@ -0,0 +1,55 @@
+# Admin-AI Model Pricing & Selection Guide (Jul 10, 2026)
+
+## Available Non-OpenRouter Models (direct through admin-ai LiteLLM)
+
+These models are accessible via `https://admin-ai.itpropartner.com/v1` with the admin-ai virtual key. All 37 models in Postgres route through individual provider API keys (Anthropic, DeepSeek, Google, OpenAI, xAI, Groq) — only the single `openrouter/*` entry routes through OpenRouter.
+
+## Current Fallback Chain (configured Jul 10, requires gateway restart)
+
+| Level | Model | Provider | Input/Output $/M |
+|---|---|---|---|
+| Primary (both conversation + subagent) | deepseek-v4-pro | admin-ai | $0.435 / $0.87 |
+| Fallback 1 | deepseek-chat | admin-ai | $0.14 / $0.28 |
+| Fallback 2 | gemini-flash-latest | admin-ai | $0.08–0.30 / $0.30–2.50 |
+| Fallback 3 | llama3.2:3b | ollama-local | Free |
+
+## Full Pricing Table (non-OpenRouter models)
+
+| Model | Input $/M | Output $/M | vs deepseek-chat |
+|---|---|---|---|
+| gemini-flash-latest | $0.08–0.30 | $0.30–2.50 | ~1–2x |
+| deepseek-chat (V3) | $0.14 | $0.28 | 1x (baseline) |
+| deepseek-v4-flash | ~$0.15 | ~$0.30 | ~1x |
+| deepseek-coder | $0.14 | $0.28 | ~1x |
+| groq/qwen/qwen3-32b | ~$0.35 | ~$0.35 | ~2x |
+| deepseek-v4-pro | $0.435 | $0.87 | ~3x |
+| deepseek-reasoner (R1) | $0.55 | $2.19 | ~4–8x |
+| claude-haiku-4-5 | $1.00 | $5.00 | ~7–18x |
+| gemini-pro-latest | $1.25 | $5.00 | ~9–18x |
+| claude-sonnet-4-6 | $3.00 | $15.00 | ~21–54x |
+| claude-opus-4-8 | $5.00 | $25.00 | ~36–89x |
+| gpt-5.5 | $5.00 | $30.00 | ~36–107x |
+
+## Models Worth Adding (best price/performance ratio)
+
+| Model | Input/Output $/M | Provider | Why |
+|---|---|---|---|
+| Qwen 3.5 27B | ~$0.35 / $0.35 | Together/Groq | Best coding model per dollar |
+| Mistral Small | $0.10 / $0.30 | Mistral | Cheapest general-purpose |
+| Llama 4 70B | ~$0.60 / $0.90 | Together | 1M context, open-weight |
+
+Already available through admin-ai via Groq. Adding a Together AI account would unlock Qwen 3.5, Llama 4, Mistral models through a single provider.
+
+## Virtual Key Config (current as of Jul 10)
+
+The entire model chain now uses a virtual key (`sk-mTvC...`) with $50/mo budget and model restrictions, not the master key. Key management via `https://admin-ai.itpropartner.com/ui`.
+
+## Model Name Format (CRITICAL)
+
+Use the EXACT name admin-ai exposes — hyphens, no OpenRouter prefix:
+- `claude-opus-4-8` — direct through admin-ai
+- `openrouter/anthropic/claude-opus-4.8` — routes through OpenRouter, hits credit wall
+
+## Gateway Restart Required
+
+After changing model config, gateway must restart. No hot-reload for delegation model config.
@@ -0,0 +1,137 @@
+# Model Routing Policy — 2026-07-14
+
+## User-confirmed routing shape
+
+Production model routing must follow this provider order:
+
+```text
+Primary     → admin-ai
+Fallback 1  → admin-ai
+Fallback 2  → OpenRouter direct
+Fallback 3  → independent direct frontier-capable provider
+```
+
+At Fallback 3, the priority is **uptime/access to Sho'Nuff and model competence**, not cost optimization. The model must be capable enough for serious ops/debugging/infrastructure work, not merely available.
+
+## Concrete target route from this session
+
+```text
+Primary:
+  deepseek-v4-pro via admin-ai
+
+Fallback 1:
+  deepseek-chat via admin-ai
+
+Fallback 2:
+  openrouter/qwen/qwen3.6-plus via OpenRouter direct
+
+Fallback 3:
+  xai/grok-4.3 via xAI direct
+```
+
+This satisfies:
+
+- admin-ai primary control and LiteLLM logging
+- same-router model fallback
+- independent OpenRouter fallback
+- independent direct frontier-capable fallback
+- no Ollama/Gemini in normal text routing
+
+## Fallback 3 provider pool
+
+Germaine stated keys are available for:
+
+- OpenAI
+- Perplexity
+- Groq
+- OpenRouter
+- DeepSeek
+- xAI
+- Mistral
+- Fireworks AI
+- Anthropic
+- Gemini
+- Cohere
+- AI21
+- Deepgram
+
+For Fallback 3:
+
+- Exclude OpenRouter because it is already Fallback 2.
+- Exclude Deepgram from LLM routing; it is speech/audio, not a general chat brain.
+- Exclude Gemini from normal text routing unless Germaine explicitly re-approves it.
+- Prefer frontier-capable direct providers over cheap/fast small models.
+
+Recommended direct-provider selection order, subject to live completion and quota tests:
+
+1. OpenAI direct GPT-5.5/GPT-5-class, if quota/billing healthy
+2. Anthropic direct Claude Opus/Sonnet-class, if usage limit healthy
+3. xAI direct high-end Grok
+4. Mistral direct top model, if configured and tested
+5. Fireworks AI high-end hosted model, if selected and tested
+6. Groq direct only if the selected model is good enough for Sho'Nuff ops
+7. DeepSeek direct only if frontier providers are unavailable and uptime is at risk
+8. Cohere / AI21 only after explicit quality testing
+
+## Direct-vs-aggregator distinction
+
+Model IDs prefixed with `openrouter/` are OpenRouter routes even if the model family is OpenAI, Anthropic, Qwen, etc. Examples:
+
+```text
+openrouter/openai/gpt-5.2-pro         → OpenRouter route
+openrouter/anthropic/claude-opus-4.7  → OpenRouter route
+openrouter/qwen/qwen3.6-plus          → OpenRouter route
+```
+
+A true direct provider fallback should use that provider's direct route, e.g. `xai/grok-4.3` for xAI direct.
+
+## Verified candidates from this session
+
+Working via current app1 admin-ai endpoint:
+
+```text
+deepseek-v4-pro                      OK
+deepseek-chat                        OK
+xai/grok-4.3                         OK
+xai/grok-4-1-fast                    OK
+openrouter/qwen/qwen3.6-plus         OK via OpenRouter route
+openrouter/openai/gpt-5.2-pro        OK via OpenRouter route
+openrouter/anthropic/claude-opus-4.7 OK via OpenRouter route
+```
+
+Failing/limited during this session:
+
+```text
+gpt-5.5 direct/admin-ai              OpenAI insufficient_quota
+claude-opus/sonnet direct            Anthropic usage limit until 2026-08-01
+mistral direct names tested          no healthy LiteLLM deployment configured
+qwen/qwen3-32b direct name           no healthy LiteLLM deployment configured
+```
+
+## Explicit exclusions from automatic routing
+
+- Ollama / `llama3.2:3b`: too weak for Sho'Nuff operations; do not use as automatic fallback or LLM cron model.
+- Gemini: removed from normal model-routing plan; do not reintroduce as normal fallback just because it tests cleanly. It may remain available for vision/manual use if approved.
+- GPT/Claude: escalation/frontier candidates only when quotas allow; do not make them hidden uptime dependencies while quota-limited.
+
+## Health check requirements
+
+Do not trust `/v1/models` alone. A route is healthy only if a real `/v1/chat/completions` call returns usable content.
+
+Check each production tier separately:
+
+- admin-ai primary model completion
+- admin-ai fallback model completion
+- OpenRouter direct fallback completion
+- selected Fallback 3 direct-provider completion
+
+Use enough `max_tokens` for reasoning models; too-small budgets can produce blank/length-limited responses and false negatives.
+
+GPT/Claude quota failures should be reported as escalation-model unavailable, not as total admin-ai failure.
+
+## Background jobs
+
+- Script-only jobs should stay `no_agent=true` where possible.
+- No cron/background job should be pinned to Ollama.
+- Jobs using admin-ai must use admin-ai model IDs, e.g. `deepseek-chat`, not OpenRouter model IDs like `deepseek/deepseek-chat`.
+- Pinned cron jobs do not inherit the conversation fallback chain; audit and pin them explicitly.
@@ -0,0 +1,36 @@
+# Ollama Model RAM Sizing — Core Server
+
+## Server Specs
+- **Model:** netcup RS 2000 G12
+- **RAM:** 15 GB total
+- **vCPUs:** 8 (EPYC 9645)
+- **Swap:** /dev/vzswap (compressed swap — not real RAM)
+
+## Model Testing Results (Jul 11, 2026)
+
+| Model | Params | Quant | Size on Disk | Loads? | RAM Needed | Notes |
+|-------|--------|-------|-------------|--------|------------|-------|
+| `llama3.2:3b` | 3B | Q4_K_M | 2.0 GB | ✅ Yes | ~3-4 GB | Default local fallback |
+| `llama3.3:70b` | 70B | Q4_K_M | 42 GB | ❌ No | ~35-40 GB | Error: `insufficient memory (attempted to allocate 31320 MB)` |
+
+## Diagnostic Commands
+
+```bash
+# Show currently LOADED models (models actually in memory)
+ollama ps
+
+# Show DOWNLOADED models (models available but may not load)
+ollama list
+
+# Check memory usage
+free -h
+
+# Check Ollama errors
+journalctl -u ollama --no-pager | grep -i "insufficient memory\|failed to load"
+```
+
+## Key Rule
+
+`ollama list` shows what's downloaded. `ollama ps` shows what's actually loadable. A model can pass `ollama list` and still be unusable if RAM is insufficient. Always check `ollama ps` + `journalctl -u ollama` before declaring a model operational.
+
+For this server (15 GB RAM), the maximum viable model is ~7B params at Q4 quantization (~4-5 GB RAM). 13B models at Q4 may fit (~8 GB) but leave little room for Ollama's overhead + system processes. 70B models are impossible without additional RAM.
@@ -0,0 +1,22 @@
+# Per-Profile Model Configuration
+
+## Default Profile (Sho'Nuff)
+- Config: /root/.hermes/config.yaml
+- Main model: deepseek-chat
+- Vision: gemini-pro-latest (set Jul 9, 2026 — was openrouter/openai/gpt-4o)
+- Delegation: claude-opus-4.8 (admin-ai direct, NOT OpenRouter)
+- Fallback 1: claude-opus-4.8 (admin-ai)
+- Fallback 2: llama3.2:3b (ollama-local)
+
+## Anita Profile
+- Config: /root/.hermes/profiles/anita/config.yaml
+- Main model: deepseek-v4-pro
+- Vision: gemini-pro-latest (set Jul 9, 2026 independently)
+- Fallback: llama3.2:3b (ollama-local) (set Jul 9, 2026 — previously none)
+
+## Key Facts
+1. Profiles are FULLY INDEPENDENT. Changes to default do NOT propagate to Anita.
+2. `hermes config set` modifies the current profile's config. For Anita: `hermes config set --profile anita ...`
+3. Vision model is `auxiliary.vision.model` not `vision.model`.
+4. Both profiles use the same admin-ai api_key so all models available to one are available to both.
+5. A profile WITHOUT a fallback configured has no redundancy — if admin-ai goes down, that profile is dead.
@@ -0,0 +1,38 @@
+# Systematic AI Provider Key Testing
+
+## Test protocol (2026-07-15 session)
+
+When given multiple provider API keys to verify, test each one systematically:
+
+### Step 1 — List models (lightweight, no cost)
+```bash
+curl -sS https://api.<provider>.com/v1/models \
+  -H "Authorization: Bearer <key>" --max-time 10
+```
+Confirms: key is valid, provider is reachable, models are available.
+
+### Step 2 — Chat completion (real cost)
+```bash
+curl -sS -X POST https://api.<provider>.com/v1/chat/completions \
+  -H "Authorization: Bearer <key>" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"<smallest-model>","messages":[{"role":"user","content":"Say OK if working"}],"max_tokens":10}'
+```
+Use the smallest/cheapest model. Confirms: completions actually work, no billing/tier blocks.
+
+### Step 3 — Classify result
+| Result | Meaning |
+|---|---|
+| Models 200 + Completions 200 | ✅ Fully operational |
+| Models 200 + Completions 4xx | ⚠️ Key valid but billing/tier blocks completions |
+| Models 401 | ❌ Key invalid/expired |
+| Completions 200 but model wrong | ⚠️ List models first, use exact model ID |
+
+### Gotcha — model name mismatch
+Some APIs return model IDs that differ from documentation. Always `GET /models` first and use the EXACT `id` field, never assume model names.
+
+### Storage
+All keys → `/root/.hermes/.env` (chmod 600), one `PROVIDER_API_KEY=<value>` per line.
+
+## AI21-specific gotcha (Jul 2026)
+AI21 API keys come in two formats — the user key prefix may differ from what works. The original key returned "Authentication required" while a different key format worked. Always test immediately after receiving a key, don't assume one format works.