hermes-skills/skills/devops/model-failover-and-credit-tracking/references/litellm-https-spend-endpoints.md

LiteLLM HTTPS Spend Query Endpoints (Fallback for SSH)

When SSH to the admin-ai server is unavailable (key not on this box, server temporarily unreachable), spend data can be queried directly from the LiteLLM proxy's HTTPS REST API at https://admin-ai.itpropartner.com.

Auth

Extract the admin-ai API key from Hermes config:

KEY=$(grep -A2 'admin-ai:' ~/.hermes/config.yaml | grep api_key | awk '{print $2}')

Endpoints

/global/spend — Total cumulative spend across all keys

curl -s https://admin-ai.itpropartner.com/global/spend \
  -H "Authorization: Bearer $KEY"

Returns: {"spend": 228.20, "max_budget": 0.0}

max_budget: 0.0 means unlimited (no budget cap set).

/spend/keys — Per-key cumulative spend + metadata

curl -s https://admin-ai.itpropartner.com/spend/keys \
  -H "Authorization: Bearer $KEY"

Returns JSON array where each entry has:

• key_name — last-4-char suffix (e.g. sk-...bWPA)
• key_alias — human-readable alias (e.g. "Hermes", "agent.iamgmb.com")
• spend — cumulative dollar amount
• last_active — ISO timestamp of last usage
• models — list of models this key has access to
• max_budget — budget cap (null = none)
• created_at / updated_at — timestamps

/spend/tags — Spend broken down by credential/tag

curl -s https://admin-ai.itpropartner.com/spend/tags \
  -H "Authorization: Bearer $KEY"

Returns JSON array where each entry has:

• individual_request_tag — e.g. "Credential: Deepseek", "User-Agent: OpenAI/Python 2.24.0"
• total_spend — cumulative dollars
• log_count — number of requests

Limitations

• These are cumulative lifetime figures. For 24h-only data, use the SSH+Postgres query (the LiteLLM_SpendLogs table has timestamps).
• The HTTPS API does not expose per-model spend breakdown for the last 24h — only cumulative per-key and per-tag.
• The OpenRouter API key in config.yaml typically returns 401 "User not found" when queried — the stored key is dead/expired. OpenRouter credit balance cannot be queried from this box. State as "Unavailable" in reports. Check manually at https://openrouter.ai/settings/credits.
• Cron agent analysis cannot use these HTTPS endpoints. The cron_mode: deny setting blocks curl-pipe patterns (tirith blocks curl | python3) and web_extract can't send auth headers. These endpoints are useful only for interactive sessions or the pre-run collection script.

Data Cross-Check

The /spend/keys cumulative per-key figures match the Postgres LiteLLM_VerificationToken table. The /global/spend total may differ slightly from the sum of per-key spends because /global/spend is the proxy's internal counter (slightly behind real-time), while /spend/keys reads from the DB directly.