Engineering write-up on building a provider-agnostic LLM gateway: why to define the configuration contract before the abstraction, how a single generic adapter covers most vendors, why a time reserve is required so a slow primary does not starve every retry, why the model allowlist must be enforced server-side, and how reasoning models can return an empty but successful response unless reasoning output is explicitly disabled.
Shipping a Provider-Agnostic LLM Gateway
If your product calls exactly one model provider, that provider's incident page is your status page. Making the managed path swappable took less code than expected. Most of the lessons were about failures that look like success — a slow primary that starves every fallback, a reasoning model that answers with an empty HTTP 200, a blank setting that quietly disables a provider — plus the server-side guardrails that keep model selection from becoming a spend vector.
The Contract Comes First
The instinct is to write an abstraction layer. The more useful first step is to define the configuration surface, because that is what forces the abstraction to stay honest.
Write down what an operator is allowed to set — which provider leads, which model it serves, which alternatives stand behind it — before writing any adapter code. Every abstraction decision after that gets checked against one question: can it still be expressed in that contract?
If you build a sequence like this, one rule matters more than the rest. Every attempt should draw from a single shared time budget, with a reserve held back so the last alternative still has room to answer rather than inheriting the scraps of the window.
That rule is the one that gets discovered late. Without a reserve, a slow primary consumes the entire request window and every fallback fails on timeout — which reads in logs as "all providers are down."
One Adapter Covers Most Providers
A few vendors need a dedicated adapter because their request shape is genuinely their own. Nearly everything else speaks a common, widely adopted request dialect closely enough that one generic adapter — parameterised by base URL, model name, and API key — covers the rest.
This is where the temptation to over-abstract appears. Resist adding a plugin registry. A frozen record of provider configurations, typed as a closed union, gives compile-time exhaustiveness and makes an unsupported provider a type error instead of a runtime one.
That closed union is also what makes retiring a vendor safe. Swapping one out becomes a compile error at every site that still refers to it, instead of a dead string discovered in production weeks later.
Reasoning Quirks That Fail Silently
Reasoning models default to spending their output budget on internal traces. When that budget is exhausted the response is technically successful and completely empty. No error, no warning, HTTP 200, and nothing in the content field. Even below that limit, every request pays the full reasoning latency on a request window that assumes a fast model.
The fix is to disable reasoning output explicitly, and that is where the quirks live. Each vendor spells the switch differently — a different key, a different nesting level, sometimes a different key for each model generation. An outdated key is not rejected; it is silently ignored, so a stale flag looks correct in review while every request still pays for reasoning.
Omit or misspell the switch and the provider looks healthy while returning nothing, or answering far too slowly. Check the content, not the status code: an empty answer with HTTP 200 is a failure and should be handled as one. The switches belong in the adapter, not in call sites, and each one deserves a comment explaining it is mandatory — the natural instinct on encountering them later is to delete them as dead configuration.
Lock the Model Allowlist Server-Side
Letting clients name a model is a useful feature and a spend vector. If the model string is accepted as-is, any client can point every request at your most expensive model.
The fix is a single shared allowlist consumed by both the client picker and a server-side schema, so an unlisted model is rejected at the boundary. Two adjacent decisions matter as much:
An explicit model selection should pin its owning provider. Selecting a model but routing to whichever provider happens to be primary produces a confusing class of bug where the response does not match the requested model.
An explicit model selection should also bypass a stored bring-your-own-key credential. Otherwise a user who picks a managed model can silently have that request routed to their own custom endpoint, which is neither what they asked for nor what the billing implies.
What Blank Environment Variables Taught Us
An unset environment variable and a blank one are not the same thing, and the difference is easy to miss. Depending on the platform and how the value was edited, an "unset" variable can arrive as an empty string rather than as undefined. A schema default that only applies to undefined will happily accept "" and skip the default, which means a blank variable silently disables a provider rather than falling back to the intended value.
Normalise empty strings to undefined in the schema preprocessor. This is a two-line change that prevents a category of misconfiguration that is otherwise invisible until traffic arrives.
The Result
The managed path no longer depends on a single vendor. When the active provider fails, the request moves to an alternative instead of failing with it. Adding or retiring a vendor is now a configuration change plus whatever edits the compiler demands, not a migration — which is the clearest signal that the abstraction actually holds.