Routing and failover for agents

Every provider fails sometimes. On llm.buzz a model can be served by more than one provider, so a failure at one of them is a reason to try the next, not a reason to fail your agent’s turn. This guide covers the order providers are tried in, what moves a request on, and the fields your agent can use to steer it.

Models with the most providers

Failover needs somewhere to go. Right now 43 models on llm.buzz have more than one live provider.

From the public models list, read 27 October 2026, 05:52 UTC.

A provider named llm.buzz is llm.buzz’s own route to a model, priced at the lowest of the model’s direct providers’ prices, price by price, and it takes part in failover like any provider. The llm.buzz provider explains it.

The order is set once

Each model has a routing preset that orders its providers:

A suffix on the model’s name changes the preset for one request: :floor puts the cheapest provider first and :nitro the fastest. provider.sort does the same in the body: "price" works as :floor, "latency" and "throughput" as :nitro, and "smart" ranks by price and speed.

extra_body={"provider": {"sort": "price"}}

The order is fixed when the request is admitted. Failover then walks it from the top.

What moves a request to the next provider

The request goes to the first provider in the order. If it errors or times out, the next provider for the same model answers. Your agent sees one request and one answer, and pays only the provider that answered.

A provider that keeps failing is skipped for a short while, so requests in that window go straight to the next provider instead of waiting for a timeout first.

A refusal is not a failure. It comes back from the provider that refused, with gateway_code: refusal, and is not tried elsewhere.

Steer the providers

The provider object narrows or reorders the providers of one request, as on OpenRouter. It works on Chat Completions, Responses, Messages, Embeddings and the image routes; the audio routes read only its zdr and data_collection.

A name matches a provider’s display name, its tag or the tag’s first part, ignoring case. The tags are on the endpoints route, GET https://api.llm.buzz/api/v1/models/{author}/{slug}/endpoints. With an order, sort suffixes and provider.sort no longer apply.

reply = client.chat.completions.create(
    model="moonshotai/kimi-k3",
    messages=[{"role": "user", "content": "Plan the next step"}],
    extra_body={"provider": {"order": ["DeepInfra"]}},
)

To pin one provider, for a benchmark or a comparison, add "allow_fallbacks": False. Its failure is then your answer. To keep a request on the companies that serve a model directly, add "ignore": ["llm.buzz"].

When your preferences leave the model no provider, the answer is 404 no_allowed_providers, its message names the providers that serve the model, and nothing is sent. Fix the request: retrying it unchanged does not help. When the providers you allow are all down, the answer is a retryable 503 unavailable_route instead.

Bound the retries

The gateway object sets ceilings for one request on Chat Completions, Responses and Messages. Every field is optional.

extra_body={"gateway": {
    "retry": {"max_attempts_per_route": 2, "max_total_attempts": 3,
              "backoff": {"type": "exponential", "base_delay_ms": 500, "max_delay_ms": 8000}},
}}

max_attempts_per_route caps calls to one provider, max_total_attempts caps calls in all, fallbacks included, and backoff sets the wait before calling a throttled provider again. These are ceilings, not promises: the request’s time limit, your spending limits and a provider’s health can stop it sooner. Once output has reached you, the request stays on the provider that sent it.

Retry safely

When every provider tried fails, the answer is 502 all_routes_failed. Retry a 429, 502, 503 or 504 with backoff, and wait Retry-After seconds first when the answer carries it. After an error answer, send the retry with a new key: the old key answers 409 idempotency_replay_unavailable after most errors (What to send next). Do not resend any other 4xx unchanged.

On Chat Completions and Responses, send an Idempotency-Key header. A retry with the same key and the same body never runs the request twice, so a timeout on your side never runs a tool-calling turn twice: the retry returns the first answer, or 409 idempotency_replay_unavailable when that answer is not stored where the retry arrived. Reusing a key with a different body is a 409 idempotency_conflict.

On idempotency_replay_unavailable, llm.buzz accepted the turn and did not run it again. It may still be running, or its answer may exist, so retry with the same key after a short backoff first. Do not send the turn again without the key or with a new one unless running it twice is acceptable, or the first call ended in an error answer, which leaves its key answering 409 for good: a new key runs the turn again and bills it again. What to send next has the cases.

reply = client.chat.completions.create(
    model="moonshotai/kimi-k3",
    messages=[{"role": "user", "content": "Plan the next step"}],
    extra_headers={"Idempotency-Key": "run-42-turn-7"},
)

A stream is a special case. Once it has started, a failure arrives as one last chunk with an error object and finish_reason: "error", followed by [DONE]. Check for it, so your agent never takes a cut answer for a complete one.