hermes-skills/skills/devops/model-failover-and-credit-tracking/references/litellm-spend-query-patterns.md

LiteLLM Spend Query Patterns (app1 Postgres)

Quick-reference for querying today's spend directly from LiteLLM's Postgres database on app1.

SSH Access

ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131

Today's Total Spend

ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
SELECT 
  ROUND(SUM(spend)::numeric, 4) as total_spend,
  COUNT(*) as total_requests,
  SUM(total_tokens) as total_tokens
FROM \"LiteLLM_SpendLogs\" 
WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
  AND \"startTime\" < '\''2026-07-15 00:00:00'\'';
"'

Per-Model Breakdown

ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
SELECT 
  model,
  COUNT(*) as requests,
  ROUND(SUM(spend)::numeric, 4) as spend,
  SUM(total_tokens) as tokens
FROM \"LiteLLM_SpendLogs\" 
WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
GROUP BY model
ORDER BY spend DESC;
"'

Per-User/Key Breakdown

ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
SELECT 
  COALESCE(u.user_email, s.user, s.api_key) as identity,
  COUNT(*) as requests,
  ROUND(SUM(s.spend)::numeric, 4) as spend,
  SUM(s.total_tokens) as tokens
FROM \"LiteLLM_SpendLogs\" s
LEFT JOIN \"LiteLLM_UserTable\" u ON s.user = u.user_id
WHERE s.\"startTime\" >= '\''2026-07-14 00:00:00'\'' 
  AND s.\"startTime\" < '\''2026-07-15 00:00:00'\''
GROUP BY identity
ORDER BY spend DESC;
"'

Failed vs Successful Requests

ssh -i /root/.ssh/itpp-infra -o StrictHostKeyChecking=no root@152.53.36.131 \
  'docker exec litellm_postgres psql -U litellm -d litellm_db -c "
SELECT status, COUNT(*), ROUND(SUM(spend)::numeric, 4) as total_spend
FROM \"LiteLLM_SpendLogs\" 
WHERE \"startTime\" >= '\''2026-07-14 00:00:00'\'' 
  AND \"startTime\" < '\''2026-07-15 00:00:00'\''
GROUP BY status
ORDER BY count DESC;
"'

Pitfalls

• Column names are camelCase: startTime, endTime, completionStartTime — always quoted
• Table name is LiteLLM_SpendLogs (capital L in Logs)
• Database is litellm_db, not litellm
• Single-quote escaping in nested SSH+Docker+psql is tricky: use '\\'' to break out of the outer single quote, insert a literal ', then resume
• Failed requests have spend = 0 and total_tokens = 0 — filter with WHERE spend > 0 for spend-only analysis
• The viewer key (sk-dZ6QafZLLN5Ewl0NxVlRhQ) only has llm_api_routes — cannot query /global/spend or /spend/logs via HTTPS. Go through Postgres directly.
• NULL-model failure rows are external auth noise. Rows with empty model, model_group, and custom_llm_provider + status = 'failure' are requests rejected before model routing (auth failures from OpenWebUI or stale integrations). Filter them out for Hermes-specific analysis: WHERE (model != '' AND model IS NOT NULL). See references/litellm-null-model-failures.md. These rows inflate failure counts in reports — report them separately.

Column Reference

Key columns in LiteLLM_SpendLogs:

Column	Type	Notes
startTime	timestamp	Request time (server local)
spend	double precision	Cost in USD
total_tokens	integer	Total tokens consumed
prompt_tokens	integer	Input tokens
completion_tokens	integer	Output tokens
model	text	Model name as called
model_group	text	Model group (e.g., deepseek-v4-pro)
custom_llm_provider	text	Provider (e.g., deepseek, openrouter)
user	text	User ID
api_key	text	API key hash used
status	text	success or failure
cache_hit	text	true if cache hit