Systematic AI Provider Key Testing

Test protocol (2026-07-15 session)

When given multiple provider API keys to verify, test each one systematically:

Step 1 — List models (lightweight, no cost)

curl -sS https://api.<provider>.com/v1/models \
  -H "Authorization: Bearer <key>" --max-time 10

Confirms: key is valid, provider is reachable, models are available.

Step 2 — Chat completion (real cost)

curl -sS -X POST https://api.<provider>.com/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"<smallest-model>","messages":[{"role":"user","content":"Say OK if working"}],"max_tokens":10}'

Use the smallest/cheapest model. Confirms: completions actually work, no billing/tier blocks.

Step 3 — Classify result

Result	Meaning
Models 200 + Completions 200	✅ Fully operational
Models 200 + Completions 4xx	⚠️ Key valid but billing/tier blocks completions
Models 401	❌ Key invalid/expired
Completions 200 but model wrong	⚠️ List models first, use exact model ID

Gotcha — model name mismatch

Some APIs return model IDs that differ from documentation. Always GET /models first and use the EXACT id field, never assume model names.

Storage

All keys → /root/.hermes/.env (chmod 600), one PROVIDER_API_KEY=<value> per line.

AI21-specific gotcha (Jul 2026)

AI21 API keys come in two formats — the user key prefix may differ from what works. The original key returned "Authentication required" while a different key format worked. Always test immediately after receiving a key, don't assume one format works.

1.6 KiB Raw Blame History