# Admin-AI Cost and Spend Audit

Use this when selecting or validating Hermes' primary model on `admin-ai.itpropartner.com`, especially after surprising spend appears in provider dashboards.

## Key lessons

- Hermes' `admin-ai` provider is a LiteLLM proxy. A model like `gpt-5.5` can bill the OpenAI backend even though Hermes only sees `provider: admin-ai`.
- Large gateway sessions can make every turn expensive because the full prompt/context may reach 100k-270k+ prompt tokens per call.
- Always distinguish:
  - Hermes-facing provider/key: `providers.admin-ai.api_key` in `/root/.hermes/config.yaml`
  - LiteLLM backend provider/key: OpenAI/Gemini/Anthropic/DeepSeek credentials stored in LiteLLM on the admin-ai server
  - Runtime fallback chain: top-level `fallback_providers`, verified with `hermes fallback list`

## Current cost-aware model-selection workflow

1. Verify the fallback chain before changing the primary:
   ```bash
   hermes fallback list
   ```
   Required survival chain:
   - `deepseek/deepseek-chat` via `openrouter`
   - `llama3.2:3b` via `ollama-local`

2. Query recent LiteLLM spend by model group on the admin-ai server:
   ```bash
   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
     \"SELECT COALESCE(model_group,''), COALESCE(custom_llm_provider,''), COUNT(*), ROUND(SUM(spend)::numeric,6), SUM(prompt_tokens), SUM(completion_tokens), SUM(total_tokens), ROUND((SUM(spend)/NULLIF(SUM(total_tokens),0)*1000000)::numeric,6) AS effective_usd_per_mtok, MAX(\\\"startTime\\\") FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= NOW() - INTERVAL '72 hours' AND model_group <> '' GROUP BY model_group, custom_llm_provider HAVING SUM(total_tokens) > 10000 ORDER BY effective_usd_per_mtok ASC NULLS LAST;\""
   ```

3. For same-day spend matching provider dashboards, group by session/user to find context size issues:
   ```bash
   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
     \"SELECT COALESCE(session_id,'') as session, COALESCE(\\\"user\\\",'') as \\\"user\\\", model_group, COUNT(*), ROUND(SUM(spend)::numeric,4), SUM(prompt_tokens), SUM(completion_tokens) FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= DATE_TRUNC('day', NOW()) GROUP BY session_id, \\\"user\\\", model_group ORDER BY SUM(spend) DESC NULLS LAST LIMIT 20;\""
   ```

4. Verify candidate model exists and can answer before switching:
   ```bash
   KEY=$(python3 - <<'PY'
   import yaml
   c=yaml.safe_load(open('/root/.hermes/config.yaml'))
   print(c['providers']['admin-ai']['api_key'])
   PY
   )
   curl -sS https://admin-ai.itpropartner.com/v1/chat/completions \
     -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     -d '{"model":"gemini-pro-latest","messages":[{"role":"user","content":"Return exactly OK."}],"max_tokens":120}'
   ```
   Do not use too-low `max_tokens` with Gemini; it can spend reasoning tokens and return `content: null`.

5. Make one config change at a time and verify:
   ```bash
   cp -a /root/.hermes/config.yaml /root/.hermes/config.yaml.bak.$(date +%Y%m%d-%H%M%S)-pre-model-change
   hermes config set model.default gemini-pro-latest
   hermes config set model.provider admin-ai
   hermes fallback list
   hermes config show
   ```

## Cost notes from observed usage

Observed effective costs vary by route/model alias. Prefer measured admin-ai spend over assumptions.

Examples observed in LiteLLM spend logs:

| Model group | Backend | Effective cost signal |
|---|---|---|
| `deepseek/deepseek-v4-flash` | DeepSeek | very low; good cheap default candidate |
| `deepseek-v4-pro` | DeepSeek | low; better than `deepseek-chat` and cost-effective |
| `gemini-pro-latest` | Gemini | mid-cost; more competent than DeepSeek Chat, far cheaper than GPT/Claude in this setup |
| `gpt-5.5` | OpenAI | can become expensive fast in large-context Telegram sessions |
| `claude-sonnet-4-6` | Anthropic | strong but expensive for default use |

## Backend key attribution

When the user sees spend in Google AI Studio or platform.openai.com:

- Query `LiteLLM_SpendLogs` grouped by `model_group`, `custom_llm_provider`, and time range.
- `api_key` in `LiteLLM_SpendLogs` is the LiteLLM virtual/master key hash used by Hermes, not necessarily the raw upstream provider key.
- Google/Gemini backend credentials may live encrypted in `LiteLLM_CredentialsTable` and model rows may reference encrypted `litellm_credential_name`; do not claim exact raw key mapping unless you have explicitly decrypted/verified it.

Useful table/column facts:

- Spend table: `"LiteLLM_SpendLogs"`
- Time column is quoted camelCase: `"startTime"`
- Model table: `"LiteLLM_ProxyModelTable"`
- Credential table: `"LiteLLM_CredentialsTable"`