Engineering write-up on building a provider-agnostic LLM gateway: why to define the configuration contract before the abstraction, how a single OpenAI-compatible adapter covers most vendors, why a time reserve is required so a slow primary does not starve every retry, why the model allowlist must be enforced server-side, and how reasoning models can return an empty but successful response unless reasoning output is explicitly disabled.
Shipping a Provider-Agnostic LLM Gateway
If your product calls exactly one model provider, that provider's incident page is your status page. Making the managed path swappable took less code than expected and produced two failure modes that were genuinely difficult to diagnose.
The Contract Comes First
The instinct is to write an abstraction layer. The more useful first step is to define the configuration surface, because that is what forces the abstraction to stay honest.
`` LLM_PROVIDER=<primary-vendor> LLM_MODEL=<primary-model> LLM_FALLBACK_CHAIN=<vendor-b>,<vendor-c>,<vendor-d>bash
`Illustrative shape, not any particular deployment
Three rules make the sequence behave predictably. A provider with no configured API key is skipped rather than attempted. The primary that just failed is never replayed inside the chain. And each attempt draws from a shared time budget, with a reserve held back so the last provider still has room to answer rather than inheriting a doomed 400 ms.
That last rule is the one that gets discovered late. Without a reserve, a slow primary consumes the entire request window and every fallback fails on timeout — which reads in logs as "all providers are down."
One Adapter Covers Most Providers
A few vendors need a dedicated adapter because their request shape is genuinely their own. Nearly everything else speaks an OpenAI-compatible dialect closely enough that one generic adapter — parameterised by base URL, model name, and API key — covers the rest.
This is where the temptation to over-abstract appears. Resist adding a plugin registry. A frozen record of provider configurations, typed as a closed union, gives compile-time exhaustiveness and makes an unsupported provider a type error instead of a runtime one.
That closed union is also what makes retiring a vendor safe. Swapping one out becomes a compile error at every site that still refers to it — the capability table, the input budgets, the egress rules — instead of a dead string discovered in production weeks later.
Two Quirks That Return Empty Answers
Reasoning models default to spending their output budget on internal traces. When that budget is exhausted the response is technically successful and completely empty. No error, no warning, HTTP 200, and nothing in the content field.
DeepSeek requires reasoning to be explicitly disabled:
`json
{ "thinking": { "type": "disabled" } }
`
NVIDIA uses a different key entirely:
`json
{ "chat_template_kwargs": { "thinking": false } }
`
Omit either and the provider looks healthy while returning nothing. Both belong in the adapter, not in call sites, and both deserve a comment explaining they are mandatory — the natural instinct on encountering them later is to delete them as dead configuration.
Lock the Model Allowlist Server-Side
Letting clients name a model is a useful feature and a spend vector. If the model string is accepted as-is, any client can redirect your managed budget to your most expensive model.
The fix is a single shared allowlist consumed by both the client picker and a server-side schema, so an unlisted model is rejected at the boundary. Two adjacent decisions matter as much:
An explicit model selection should pin its owning provider. Selecting a model but routing to whichever provider happens to be primary produces a confusing class of bug where the response does not match the requested model.
An explicit model selection should also bypass a stored bring-your-own-key credential. Otherwise a user who picks a managed model can silently have that request routed to their own custom endpoint, which is neither what they asked for nor what the billing implies.
What Blank Environment Variables Taught Us
Vercel stores an unset environment variable as an empty string, not as undefined. A Zod schema using .default() will happily accept "" and skip the default, which means a blank variable silently disables a provider rather than falling back to the intended value.
Normalise empty strings to undefined` in the schema preprocessor. This is a two-line change that prevents a category of misconfiguration that is otherwise invisible until traffic arrives.
The Result
The managed path now has no privileged provider. Any single provider can fail and requests continue to complete. The provider that was originally the only option is now bring-your-own-key only, and its API key is optional — which is the clearest signal that the abstraction actually holds.