88 lines
4.7 KiB
Markdown
88 lines
4.7 KiB
Markdown
# Admin-AI Cost and Spend Audit
|
|
|
|
Use this when selecting or validating Hermes' primary model on `admin-ai.itpropartner.com`, especially after surprising spend appears in provider dashboards.
|
|
|
|
## Key lessons
|
|
|
|
- Hermes' `admin-ai` provider is a LiteLLM proxy. A model like `gpt-5.5` can bill the OpenAI backend even though Hermes only sees `provider: admin-ai`.
|
|
- Large gateway sessions can make every turn expensive because the full prompt/context may reach 100k-270k+ prompt tokens per call.
|
|
- Always distinguish:
|
|
- Hermes-facing provider/key: `providers.admin-ai.api_key` in `/root/.hermes/config.yaml`
|
|
- LiteLLM backend provider/key: OpenAI/Gemini/Anthropic/DeepSeek credentials stored in LiteLLM on the admin-ai server
|
|
- Runtime fallback chain: top-level `fallback_providers`, verified with `hermes fallback list`
|
|
|
|
## Current cost-aware model-selection workflow
|
|
|
|
1. Verify the fallback chain before changing the primary:
|
|
```bash
|
|
hermes fallback list
|
|
```
|
|
Required survival chain:
|
|
- `deepseek/deepseek-chat` via `openrouter`
|
|
- `llama3.2:3b` via `ollama-local`
|
|
|
|
2. Query recent LiteLLM spend by model group on the admin-ai server:
|
|
```bash
|
|
ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
|
|
"docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
|
|
\"SELECT COALESCE(model_group,''), COALESCE(custom_llm_provider,''), COUNT(*), ROUND(SUM(spend)::numeric,6), SUM(prompt_tokens), SUM(completion_tokens), SUM(total_tokens), ROUND((SUM(spend)/NULLIF(SUM(total_tokens),0)*1000000)::numeric,6) AS effective_usd_per_mtok, MAX(\\\"startTime\\\") FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= NOW() - INTERVAL '72 hours' AND model_group <> '' GROUP BY model_group, custom_llm_provider HAVING SUM(total_tokens) > 10000 ORDER BY effective_usd_per_mtok ASC NULLS LAST;\""
|
|
```
|
|
|
|
3. For same-day spend matching provider dashboards, group by session/user to find context size issues:
|
|
```bash
|
|
ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
|
|
"docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
|
|
\"SELECT COALESCE(session_id,'') as session, COALESCE(\\\"user\\\",'') as \\\"user\\\", model_group, COUNT(*), ROUND(SUM(spend)::numeric,4), SUM(prompt_tokens), SUM(completion_tokens) FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= DATE_TRUNC('day', NOW()) GROUP BY session_id, \\\"user\\\", model_group ORDER BY SUM(spend) DESC NULLS LAST LIMIT 20;\""
|
|
```
|
|
|
|
4. Verify candidate model exists and can answer before switching:
|
|
```bash
|
|
KEY=$(python3 - <<'PY'
|
|
import yaml
|
|
c=yaml.safe_load(open('/root/.hermes/config.yaml'))
|
|
print(c['providers']['admin-ai']['api_key'])
|
|
PY
|
|
)
|
|
curl -sS https://admin-ai.itpropartner.com/v1/chat/completions \
|
|
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
|
|
-d '{"model":"gemini-pro-latest","messages":[{"role":"user","content":"Return exactly OK."}],"max_tokens":120}'
|
|
```
|
|
Do not use too-low `max_tokens` with Gemini; it can spend reasoning tokens and return `content: null`.
|
|
|
|
5. Make one config change at a time and verify:
|
|
```bash
|
|
cp -a /root/.hermes/config.yaml /root/.hermes/config.yaml.bak.$(date +%Y%m%d-%H%M%S)-pre-model-change
|
|
hermes config set model.default gemini-pro-latest
|
|
hermes config set model.provider admin-ai
|
|
hermes fallback list
|
|
hermes config show
|
|
```
|
|
|
|
## Cost notes from observed usage
|
|
|
|
Observed effective costs vary by route/model alias. Prefer measured admin-ai spend over assumptions.
|
|
|
|
Examples observed in LiteLLM spend logs:
|
|
|
|
| Model group | Backend | Effective cost signal |
|
|
|---|---|---|
|
|
| `deepseek/deepseek-v4-flash` | DeepSeek | very low; good cheap default candidate |
|
|
| `deepseek-v4-pro` | DeepSeek | low; better than `deepseek-chat` and cost-effective |
|
|
| `gemini-pro-latest` | Gemini | mid-cost; more competent than DeepSeek Chat, far cheaper than GPT/Claude in this setup |
|
|
| `gpt-5.5` | OpenAI | can become expensive fast in large-context Telegram sessions |
|
|
| `claude-sonnet-4-6` | Anthropic | strong but expensive for default use |
|
|
|
|
## Backend key attribution
|
|
|
|
When the user sees spend in Google AI Studio or platform.openai.com:
|
|
|
|
- Query `LiteLLM_SpendLogs` grouped by `model_group`, `custom_llm_provider`, and time range.
|
|
- `api_key` in `LiteLLM_SpendLogs` is the LiteLLM virtual/master key hash used by Hermes, not necessarily the raw upstream provider key.
|
|
- Google/Gemini backend credentials may live encrypted in `LiteLLM_CredentialsTable` and model rows may reference encrypted `litellm_credential_name`; do not claim exact raw key mapping unless you have explicitly decrypted/verified it.
|
|
|
|
Useful table/column facts:
|
|
|
|
- Spend table: `"LiteLLM_SpendLogs"`
|
|
- Time column is quoted camelCase: `"startTime"`
|
|
- Model table: `"LiteLLM_ProxyModelTable"`
|
|
- Credential table: `"LiteLLM_CredentialsTable"` |