What happens when a provider fails

Every provider fails sometimes. An endpoint times out under load, a region has a bad hour, a deploy goes wrong. If one provider is all you have, its bad hour is yours too. On llm.buzz a model is usually served by more than one provider, so a failure at one of them is a reason to try the next, not a reason to fail your request.

This post walks through what happens, in order, and what you can change.

The order is set before the first try

Each model has a routing preset that orders its providers: by price and time to first token together (smart), by the estimated cost of the request (cheapest), by recent time to first token (fastest), or in an order Surogate set (order). A suffix such as :floor or a provider.sort field can change the preset for one request.

The ordering happens once, when the request is admitted. Failover then walks that same list from the top. It never reshuffles mid-request: the next provider tried is the next one in that list, skipping any that are resting (below).

What moves a request to the next provider

The request goes to the first provider in the order. If that provider errors or times out, the next provider for the same model answers. Your code sees one request and one answer.

Not every unwelcome answer is a failure. A refusal is the model’s answer: it comes back to you, from the provider that refused, with gateway_code: refusal. So does a provider’s moderation verdict. Trying another provider would only repeat the question somewhere else, so llm.buzz does not.

A provider that fails twice in a row is skipped for 30 seconds. Requests in that window go straight to the next provider instead of paying for a timeout first, and the provider is tried again once the window ends.

Steering failover yourself

The provider object narrows or reorders the providers of one request. order names the providers to try first, in that order; the rest follow in the preset’s order:

extra_body={"provider": {"order": ["deepinfra"]}}

Sometimes you want the opposite of resilience. You are benchmarking one provider, or comparing two, and an answer from anyone else would be wrong. Setting allow_fallbacks to false stops moving to other providers:

extra_body={"provider": {"order": ["deepinfra"], "allow_fallbacks": False}}

With an order, only the listed providers are tried, in the listed order, so a failure still moves on within your list but never leaves it. Without an order, only the first provider of the preset is tried, and its failure is your answer.

only keeps just the providers you name and ignore removes some. A name matches a provider’s display name, its tag or the first part of its tag, ignoring case. The tags are on the endpoints route, GET https://api.llm.buzz/api/v1/models/{author}/{slug}/endpoints.

When your preferences leave nobody

If only, ignore or order leave the model without a provider, you get a 404 straight away, and nothing is sent to any provider:

{
  "error": {
    "code": 404,
    "message": "No allowed providers are available for the selected model. Providers serving MODEL: A, B. Adjust provider.only, provider.ignore or provider.order, or remove them.",
    "metadata": {
      "error_type": "not_found",
      "gateway_code": "no_allowed_providers",
      "param": "provider"
    }
  }
}

This is a client error, and the fix is in the request: a typo in a provider name, or a provider that does not serve this model. Retrying unchanged will not help.

It is different from a provider you named being down. When every provider your preferences allow is unavailable, the answer is a retryable 503 unavailable_route, because the request was fine and the providers were not.

When every provider fails

If every provider tried fails, the answer is 502 all_routes_failed. Retry a 429, 502, 503 or 504 with backoff, and wait Retry-After seconds first when the answer carries it. Do not resend any other 4xx unchanged.

reply = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"Idempotency-Key": "order-4711-summary"},
)

On Chat Completions and Responses, send an Idempotency-Key header so a retry after a timeout on your side does not run the request twice. The retry returns the first answer, or 409 idempotency_replay_unavailable when that answer is not stored where the retry arrived; then retry with the same key after a short backoff.

A stream is a special case. Once a stream has started, a failure arrives as one last chunk with an error object and finish_reason: "error", followed by [DONE]. Check for it, so a cut answer is not mistaken for a complete one.

What you pay for

Only the provider that answered. An attempt that failed over costs nothing, and the request is billed at the price of the provider that served it. If the second provider in the order is dearer than the first, you pay the second one’s price, because that is who did the work.

That is the point of the whole design: a provider’s bad hour should cost you a little latency, not an error and not a bill.