hermes-skills/skills/devops/model-failover-and-credit-tracking/references/admin-ai-cost-and-spend-audit.md

Admin-AI Cost and Spend Audit

Use this when selecting or validating Hermes' primary model on admin-ai.itpropartner.com, especially after surprising spend appears in provider dashboards.

Key lessons

• Hermes' admin-ai provider is a LiteLLM proxy. A model like gpt-5.5 can bill the OpenAI backend even though Hermes only sees provider: admin-ai.
• Large gateway sessions can make every turn expensive because the full prompt/context may reach 100k-270k+ prompt tokens per call.
• Always distinguish:
  • Hermes-facing provider/key: providers.admin-ai.api_key in /root/.hermes/config.yaml
  • LiteLLM backend provider/key: OpenAI/Gemini/Anthropic/DeepSeek credentials stored in LiteLLM on the admin-ai server
  • Runtime fallback chain: top-level fallback_providers, verified with hermes fallback list

Current cost-aware model-selection workflow

1. Verify the fallback chain before changing the primary:

   hermes fallback list
   

   Required survival chain:

   • deepseek/deepseek-chat via openrouter
   • llama3.2:3b via ollama-local

2. Query recent LiteLLM spend by model group on the admin-ai server:

   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
     \"SELECT COALESCE(model_group,''), COALESCE(custom_llm_provider,''), COUNT(*), ROUND(SUM(spend)::numeric,6), SUM(prompt_tokens), SUM(completion_tokens), SUM(total_tokens), ROUND((SUM(spend)/NULLIF(SUM(total_tokens),0)*1000000)::numeric,6) AS effective_usd_per_mtok, MAX(\\\"startTime\\\") FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= NOW() - INTERVAL '72 hours' AND model_group <> '' GROUP BY model_group, custom_llm_provider HAVING SUM(total_tokens) > 10000 ORDER BY effective_usd_per_mtok ASC NULLS LAST;\""
   

3. For same-day spend matching provider dashboards, group by session/user to find context size issues:

   ssh -i /root/.ssh/itpp-infra root@178.156.167.181 \
     "docker exec litellm_postgres psql -U litellm -d litellm_db -At -F '|' -c \
     \"SELECT COALESCE(session_id,'') as session, COALESCE(\\\"user\\\",'') as \\\"user\\\", model_group, COUNT(*), ROUND(SUM(spend)::numeric,4), SUM(prompt_tokens), SUM(completion_tokens) FROM \\\"LiteLLM_SpendLogs\\\" WHERE \\\"startTime\\\" >= DATE_TRUNC('day', NOW()) GROUP BY session_id, \\\"user\\\", model_group ORDER BY SUM(spend) DESC NULLS LAST LIMIT 20;\""
   

4. Verify candidate model exists and can answer before switching:

   KEY=$(python3 - <<'PY'
   import yaml
   c=yaml.safe_load(open('/root/.hermes/config.yaml'))
   print(c['providers']['admin-ai']['api_key'])
   PY
   )
   curl -sS https://admin-ai.itpropartner.com/v1/chat/completions \
     -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     -d '{"model":"gemini-pro-latest","messages":[{"role":"user","content":"Return exactly OK."}],"max_tokens":120}'
   

   Do not use too-low max_tokens with Gemini; it can spend reasoning tokens and return content: null.

5. Make one config change at a time and verify:

   cp -a /root/.hermes/config.yaml /root/.hermes/config.yaml.bak.$(date +%Y%m%d-%H%M%S)-pre-model-change
   hermes config set model.default gemini-pro-latest
   hermes config set model.provider admin-ai
   hermes fallback list
   hermes config show
   

Cost notes from observed usage

Observed effective costs vary by route/model alias. Prefer measured admin-ai spend over assumptions.

Examples observed in LiteLLM spend logs:

Model group	Backend	Effective cost signal
deepseek/deepseek-v4-flash	DeepSeek	very low; good cheap default candidate
deepseek-v4-pro	DeepSeek	low; better than deepseek-chat and cost-effective
gemini-pro-latest	Gemini	mid-cost; more competent than DeepSeek Chat, far cheaper than GPT/Claude in this setup
gpt-5.5	OpenAI	can become expensive fast in large-context Telegram sessions
claude-sonnet-4-6	Anthropic	strong but expensive for default use

Backend key attribution

When the user sees spend in Google AI Studio or platform.openai.com:

• Query LiteLLM_SpendLogs grouped by model_group, custom_llm_provider, and time range.
• api_key in LiteLLM_SpendLogs is the LiteLLM virtual/master key hash used by Hermes, not necessarily the raw upstream provider key.
• Google/Gemini backend credentials may live encrypted in LiteLLM_CredentialsTable and model rows may reference encrypted litellm_credential_name; do not claim exact raw key mapping unless you have explicitly decrypted/verified it.

Useful table/column facts:

• Spend table: "LiteLLM_SpendLogs"
• Time column is quoted camelCase: "startTime"
• Model table: "LiteLLM_ProxyModelTable"
• Credential table: "LiteLLM_CredentialsTable"