Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,104 @@
+# LiteLLM Spend Query Patterns (app1 Postgres)
+
+Quick-reference for querying today's spend directly from LiteLLM's Postgres database on app1.
+
+## SSH Access
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131
+```
+
+## Today's Total Spend
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  ROUND(SUM(spend)::numeric, 4) as total_spend,
+  COUNT(*) as total_requests,
+  SUM(total_tokens) as total_tokens
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\'';
+"'
+```
+
+## Per-Model Breakdown
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  model,
+  COUNT(*) as requests,
+  ROUND(SUM(spend)::numeric, 4) as spend,
+  SUM(total_tokens) as tokens
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY model
+ORDER BY spend DESC;
+"'
+```
+
+## Per-User/Key Breakdown
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT 
+  COALESCE(u.user_email, s.user, s.api_key) as identity,
+  COUNT(*) as requests,
+  ROUND(SUM(s.spend)::numeric, 4) as spend,
+  SUM(s.total_tokens) as tokens
+FROM \"LiteLLM_SpendLogs\" s
+LEFT JOIN \"LiteLLM_UserTable\" u ON s.user = u.user_id
+WHERE s.\"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND s.\"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY identity
+ORDER BY spend DESC;
+"'
+```
+
+## Failed vs Successful Requests
+
+```bash
+ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
+  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
+SELECT status, COUNT(*), ROUND(SUM(spend)::numeric, 4) as total_spend
+FROM \"LiteLLM_SpendLogs\" 
+WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
+  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
+GROUP BY status
+ORDER BY count DESC;
+"'
+```
+
+## Pitfalls
+
+- **Column names are camelCase**: `startTime`, `endTime`, `completionStartTime` — always quoted
+- **Table name is `LiteLLM_SpendLogs`** (capital L in Logs)
+- **Database is `litellm_db`**, not `litellm`
+- **Single-quote escaping** in nested SSH+Docker+psql is tricky: use `'\\''` to break out of the outer single quote, insert a literal `'`, then resume
+- **Failed requests have `spend = 0` and `total_tokens = 0`** — filter with `WHERE spend > 0` for spend-only analysis
+- **The viewer key (`sk-dZ6QafZLLN5Ewl0NxVlRhQ`) only has `llm_api_routes`** — cannot query `/global/spend` or `/spend/logs` via HTTPS. Go through Postgres directly.
+- **NULL-model failure rows are external auth noise.** Rows with empty `model`, `model_group`, and `custom_llm_provider` + `status = 'failure'` are requests rejected before model routing (auth failures from OpenWebUI or stale integrations). Filter them out for Hermes-specific analysis: `WHERE (model != '' AND model IS NOT NULL)`. See `references/litellm-null-model-failures.md`. These rows inflate failure counts in reports — report them separately.
+
+## Column Reference
+
+Key columns in `LiteLLM_SpendLogs`:
+
+| Column | Type | Notes |
+|---|---|---|
+| `startTime` | timestamp | Request time (server local) |
+| `spend` | double precision | Cost in USD |
+| `total_tokens` | integer | Total tokens consumed |
+| `prompt_tokens` | integer | Input tokens |
+| `completion_tokens` | integer | Output tokens |
+| `model` | text | Model name as called |
+| `model_group` | text | Model group (e.g., `deepseek-v4-pro`) |
+| `custom_llm_provider` | text | Provider (e.g., `deepseek`, `openrouter`) |
+| `user` | text | User ID |
+| `api_key` | text | API key hash used |
+| `status` | text | `success` or `failure` |
+| `cache_hit` | text | `true` if cache hit |