🛠 LiteLLM Rotating-Fallback Proxy

What this is, and why

A single endpoint at http://localhost:4000/v1 that any OpenAI-SDK client (Roo, Kilo, ChatGPT-style apps, raw curl) can target. LiteLLM rotates through 25+ free-tier and paid LLM upstreams behind one virtual model name. When one provider hits a rate limit or auth failure, the next one in the chain serves the request transparently. The caller never has to know — or care — which upstream produced the answer.

Built on 2026-05-25 to consolidate ~14 per-provider consult tools and remove the “which key do I use today?” cognitive overhead. All keys are read from a gitignored file outside the repo; nothing is hardcoded.

Quick start

Start the proxy

bash tools/start_litellm_proxy.sh --background

Or use the slash command /startvllmp in Claude Code. It is idempotent — refuses to double-launch if already running.

Point your client at it

The proxy accepts a placeholder key because real upstream authentication happens per-provider from the keys it loaded at startup. Your client doesn't need to know any of them.

Smoke test

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer anything" \
  -H "Content-Type: application/json" \
  -d '{"model":"free-mode","messages":[{"role":"user","content":"1+1?"}],"max_tokens":40}'

Response headers tell you which upstream served the call:

x-litellm-model-api-base:    https://api.groq.com/openai/v1
x-litellm-attempted-retries: 0
x-litellm-response-cost:     0.00002
x-litellm-response-duration-ms: 384

The two modes

You select a mode by passing one of these as the model field:

Automatic fallback chain

Every group has a fallback target so a fully-cooled mode escalates rather than failing:

hybrid-model    → free-mode-large → free-mode → paid-mode
free-mode       → free-mode-large → paid-mode
free-mode-large → paid-mode-large → paid-mode → free-mode
paid-mode       → paid-mode-large → free-mode-large → free-mode
paid-mode-large → free-mode-large → paid-mode → free-mode

Plus context-window-specific fallbacks for the ContextWindowExceededError class so a prompt that overflows a small-context upstream gets re-routed to a 128k+ model automatically.

Provider inventory (verified 2026-05-25)

43 keys were tested by tools/verify_all_keys.py (re-runnable any time to re-audit). Of those, 27 are alive and wired into the rotation. The rest are listed with their status so you know what you have versus what's actively serving traffic.

Free-mode pool (active)

Paid-mode pool (active)

Excluded (with reason — re-runnable verify will resurface if status changes)

Direct-target single-provider aliases

Bypass rotation by using one of these model names — useful for testing one provider in isolation:

• cloudflare-llama — direct CF Workers AI
• nvidia-deepseek-v4-pro — direct NVIDIA NIM with DeepSeek
• openrouter-ring-1t — direct OpenRouter Ring 2.6 1T
• claude-haiku-direct — direct Anthropic Claude Haiku 4.5
• deepseek-chat-direct — direct DeepSeek native

Slash commands (Claude Code)

All commands live in .claude/commands/ and are normal markdown — feel free to edit.

Security & key management

Where the keys live

• Canonical source: ~/dbpasses.txt — gitignored, outside the repo. Owner-readable only.
• Label format: human-readable line (e.g. NVIDIA:, GROQ FREE KEY:, OPEN ROUTER API KEY) followed by the key value on the next non-blank line.
• The launcher's parser tolerates trailing whitespace on labels, skips sub-header lines (pure-uppercase identifiers like sub-label markers), skips URL annotation lines, and skips section divider lines (=====).

What the proxy never does

• Never hardcodes keys in litellm_config.yaml — only os.environ/... references.
• Never echoes keys in logs (LiteLLM sanitizes; the cooldown callback records error messages but those are upstream responses, not keys).
• Never commits the keys file (gitignored).
• Never accepts requests from outside localhost (LiteLLM binds 0.0.0.0:4000 by default — if you need network isolation, run with --host 127.0.0.1 or restrict at the OS firewall level).

Re-auditing keys

Whenever you add or rotate a key, re-run the audit:

python3 tools/verify_all_keys.py

The harness pings every provider's chat-completions endpoint with “1+1?” in parallel and reports OK / DEAD / NOFUNDS / QUOTA / NOKEY / CONFIG per key. Sends a real browser User-Agent so Cloudflare-fronted providers (Groq, Together, Cerebras, AIMLAPI, LLM7) don't return their bot-protection 403.

Smart cooldown observability

Every upstream failure goes through tools/litellm_smart_cooldown.py, a custom LiteLLM logger that:

1. Classifies the failure into a category: daily_quota_cf · monthly_quota_hf · dead_key · rate_limit · server_error · bad_request · other
2. Computes a meaningful unban time per category (CF → next UTC midnight, HF → first of next month, dead keys → 24h, transient → 300s)
3. Writes the state to /tmp/litellm_cooldown_state.json with cool_until_utc, hit count, last error message, last seen time

Inspect anytime:

jq . /tmp/litellm_cooldown_state.json

Troubleshooting

"Connection error" in Roo / Kilo

Usually means the proxy is restarting. Wait 10 seconds and retry, or check /statusvllmp to confirm PID is alive.

413 "Request body too large"

One of the upstreams in the rotation has a small per-request cap (e.g. GitHub Models gpt-4o-mini on certain tiers is 8k tokens). The proxy auto-falls back to a long-context group via the generic fallbacks: rules. If you still see the error, the large group is also exhausted — switch to paid-mode-large manually.

429 "MidStreamFallbackError"

An upstream started streaming and then hit a rate limit mid-stream. LiteLLM can't always recover from this cleanly. Usually it's Gemini hitting its 20 req/day free-tier cap. The next call will rotate to a different upstream. To make this less common, top up Gemini's billing or rely on paid-mode.

"NOKEY" in /modevllmp or verify_all_keys

Means the launcher couldn't find that provider's label in ~/dbpasses.txt. Either the label string doesn't match exactly, the key file has unexpected indentation, or the key block is structured differently than expected (e.g. label-then-sub-label-then-value). Open tools/start_litellm_proxy.sh and check the parse_key line for that provider against the actual file content.

One provider keeps cooling immediately after restart

Check jq . /tmp/litellm_cooldown_state.json for the category field:

• dead_key → regenerate the key at the provider's console
• monthly_quota_hf → wait for first of next month or pay
• daily_quota_cf → waits until UTC midnight automatically
• rate_limit → transient, should clear on its own

Want to add a new provider

1. Add the key to ~/dbpasses.txt with a clear label.
2. Add a parse_key "LABEL FROM FILE" ENV_VAR_NAME line to tools/start_litellm_proxy.sh.
3. Add an entry to tools/verify_all_keys.py for the audit harness (label, env var, test function).
4. Run the audit; if it returns OK, add a model_list entry to litellm_config.yaml under the appropriate group (free-mode if zero-cost, paid-mode if billed).
5. /restartvllmp and confirm with /modevllmp.