# Retries and idempotency

llm.buzz retries across a model's providers inside one request, and an `Idempotency-Key` makes your own retries safe. This page says what each does, and what a retry costs.

## Inside one request

A model usually has several providers. Until a provider sends its first text, tool call or refusal, a provider that fails or times out is replaced by the next one, in the order the model's routing preset gives ([How a provider is chosen](models-and-routing.md#how-a-provider-is-chosen)). llm.buzz does not call a provider again for an ordinary failure. The answer's `x-gateway-provider` names the provider that answered, and `x-gateway-route-depth` says where it stood in the order, counting from 0.

Replacing a provider stops when:

- **Output has started.** The first text, tool call or refusal fixes the provider for the rest of the request, and llm.buzz never splices another provider's output into it. A later failure ends the answer with an error, or as incomplete when the provider's stream just stops: [Streaming](streaming.md#failures).
- **The model refused.** A refusal reaches you from the provider that refused it and is not tried on another.
- **The provider rejected the request.** A provider's own 400, whose message starts `provider rejected the request:`, reaches you and is not tried on another provider.
- **A ceiling is reached.** A request makes at most 8 provider calls and runs for at most 600 seconds, and the bounds you set with [`gateway`](#the-gateway-object) apply.

What a Chat Completions, Responses or Messages request costs when providers fail:

| What happened | What you pay |
|---|---|
| A provider failed and another provider answered | The provider that answered. The failed attempt adds 0. |
| A provider billed tokens but sent no text, tool call or refusal, and another provider answered | The provider that answered. llm.buzz treats the empty answer as a failure, so it adds 0. |
| Every provider sent an empty answer | The empty turn comes back as a 200 with the header `x-gateway-warning: empty_completion`, on Chat Completions, Responses and Messages. You pay for the usage the last provider reported, or, when it reported none, an estimate of its prompt: [When a provider reports no usage](usage-and-limits.md#when-a-provider-reports-no-usage). |
| The last provider tried failed before its stream opened, and reported no usage, because it answered an error status (a 5xx, its own 400 or a 429) or never received the request | Nothing: you received nothing. The request answers an error. |
| The last provider tried finished without any text or tool output and reported no usage or only zeros | An estimate of the prompt only: llm.buzz's own count of the input tokens, and nothing for output: [When a provider reports no usage](usage-and-limits.md#when-a-provider-reports-no-usage). |
| The last provider tried timed out, failed after the request was sent, or broke off after its stream opened (a stream that stopped after its output started included), and reported no usage | What llm.buzz reserved for that attempt: [When a provider reports no usage](usage-and-limits.md#when-a-provider-reports-no-usage). The request answers an error, or ends as incomplete. |

Embeddings, Images and Audio bill a failed last attempt by their own rules, often nothing: [Embeddings](embeddings.md#billing), [Images](images.md#billing) and [Audio](audio.md#billing).

A request llm.buzz refuses before it calls any provider costs nothing. That covers a 400 for a bad field, a 401, a 404, a 402 for a spent balance, and a 503 when no provider is live.

## The gateway object

`gateway` sets bounds for one request on Chat Completions, Responses and Messages:

- `retry.max_attempts_per_route` caps the calls to one provider, 1 to 4. It is a cap and adds no retries, because a failed provider is not called again.
- `retry.max_total_attempts` caps the calls in all, 1 to 8, fallbacks included. It is 8 when you leave it out.
- `retry.backoff` sets the wait before a throttled provider is called again.
- `routing.allow_fallbacks: false` keeps the request on the first provider, as `provider.allow_fallbacks: false` does.

These are ceilings: your spending limits, the 600-second limit and a provider's health can stop a request sooner. [Retries and fallbacks per request](api-reference.md#retries-and-fallbacks-per-request) has the fields and an example. A retry with an `Idempotency-Key` must send the same `gateway` object: a changed one is a 409 `idempotency_conflict`.

## Your retries

Your own retries are requests of their own. Every one that runs is billed, and an error answer may have cost something, as the table above says.

| Answer | What to do |
|---|---|
| 429 | Wait `Retry-After` seconds, then retry. A free offer's token allowance is the exception: retrying cannot help, so choose a paid model. |
| 500, 502, 503, 504 | Retry with backoff, and after `Retry-After` seconds when the answer carries it. |
| Any other 4xx, apart from 402 `in_flight_budget_exhausted` (retry after `Retry-After`) and 409 `idempotency_replay_unavailable` ([below](#when-the-answer-is-not-stored)) | Do not resend it unchanged. Fix the field, the key or the balance first: [Errors](errors.md#retries). |

The OpenAI and Anthropic SDKs retry on their own, twice by default, after a connection error and after a 408, 409, 429 or 5xx answer. Each of their retries is a request of its own, and without an `Idempotency-Key` it runs and bills again. If you keep the SDK's retries and send an `Idempotency-Key`, the SDK resends the same key. After a dropped connection that replays safely, and after a 429 with `x-gateway-admission-refused: true` the retry runs the request once. After a 502, a 503 or a 429 `unavailable_route` without that header, its retries end in 409 `idempotency_replay_unavailable`, which is then the error you see. Set `max_retries=0` in Python or `maxRetries: 0` in TypeScript when you handle retries yourself, or when one operation must bound the work it does. Which `Idempotency-Key` to send with a retry is covered [below](#when-the-answer-is-not-stored).

## Idempotency-Key

Send an `Idempotency-Key` header and a retry with the same key and the same body never runs the request twice. Use one key for each distinct request, such as a UUID, and send the same key only when you retry that request.

| API | `Idempotency-Key` |
|---|---|
| Chat Completions | Replays the first answer. |
| Responses over HTTP | Replays the first answer. A key on the WebSocket upgrade does not apply to the frames sent over it. |
| Messages | Not read: Anthropic's API has no such header, so a retry is a new request. |
| Embeddings, Images, Audio and Decisions | Not read: a retry is a new request, served and billed again. |

```python tab="Python"
import os
import time
import uuid

from openai import APIConnectionError, APIStatusError, ConflictError, OpenAI

client = OpenAI(
    base_url="https://api.llm.buzz/api/v1",
    api_key=os.environ["LLM_BUZZ_KEY"],
    max_retries=0,
)


def summarize(text: str) -> str | None:
    key = str(uuid.uuid4())  # one key for this request and its retries
    for attempt in range(5):
        try:
            reply = client.chat.completions.create(
                model="deepseek/deepseek-v4.1-flash",
                messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}],
                max_tokens=200,
                extra_headers={"Idempotency-Key": key},
            )
            return reply.choices[0].message.content
        except (APIConnectionError, ConflictError):
            # No answer, or 409 idempotency_replay_unavailable: the request may have run.
            # Ask again with the same key.
            delay = 2**attempt
        except APIStatusError as error:
            if error.status_code not in (429, 500, 502, 503, 504):
                raise
            # An error answer ended the request, and its key cannot run it again.
            key = str(uuid.uuid4())
            delay = float(error.response.headers.get("retry-after", 2**attempt))
        if attempt < 4:
            time.sleep(delay)
    raise RuntimeError("no answer after 5 attempts")


print(summarize("HTTP clients retry with backoff so they do not add to an outage."))
```

```ts tab="TypeScript"
import OpenAI from "openai";
import { randomUUID } from "node:crypto";

const client = new OpenAI({
  baseURL: "https://api.llm.buzz/api/v1",
  apiKey: process.env.LLM_BUZZ_KEY,
  maxRetries: 0,
});
const key = randomUUID(); // one key for this request and its retries
const reply = await client.chat.completions.create(
  {
    model: "deepseek/deepseek-v4.1-flash",
    messages: [{ role: "user", content: "Summarize in one sentence: HTTP clients retry with backoff." }],
    max_tokens: 200,
  },
  { headers: { "Idempotency-Key": key } },
);
console.log(reply.choices[0].message.content);
```

```bash tab="curl"
KEY=$(uuidgen)
for attempt in 1 2; do
  curl -s https://api.llm.buzz/api/v1/chat/completions \
    -H "Authorization: Bearer $LLM_BUZZ_KEY" \
    -H "Idempotency-Key: $KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "deepseek/deepseek-v4.1-flash",
      "max_tokens": 200,
      "messages": [{"role": "user", "content": "Summarize in one sentence: HTTP clients retry with backoff."}]
    }' | jq -r '.id, .usage.cost'
done
```

The curl loop prints the same `id` and `cost` twice: the second call is the stored answer of the first.

What a retry gets:

- **After the first call answered:** the stored answer, with its original `cost` and `x-request-id`. No provider is called and nothing is billed. A stream comes back as the same stream.
- **While the first call is still running:** the retry waits for it and returns its answer. It does not start a second generation.
- **After you closed a keyed stream:** the request keeps running and is billed when it finishes, and a retry with the key joins it or gets its answer. A Chat Completions, Responses or Messages stream without a key stops reading from the provider when you close it.
- **With a different request:** 409 `idempotency_conflict`. Any change to the body counts, including `stream`, the `gateway` object, a `provider` hint and a model suffix such as `:floor`. A changed `stream_options.include_usage` does not count. Send a new key for each distinct request.
- **Under another data-policy demand:** a key replays only for a call that makes the same data-policy demand as the first one did: no demand, zero retention, or no training and no retention, whether it came from `provider.zdr`, `provider.data_collection` or your organization's data policy. The call must also run under the same residency, EU or anywhere. A retry that makes another demand is not a replay: it runs and bills as a new request.
- **After a limit or a spent balance:** a retry of a request that completed still returns its stored answer once the key has reached its spending limit or your balance is spent. A new request is refused.

`x-client-request-id`, `session_id` and `x-session-id` never act as a key. Two requests with the same prompt and no `Idempotency-Key` are two requests.

## When the answer is not stored

llm.buzz keeps the answer to a keyed request for up to 24 hours, or less when it needs the room. This copy is not a prompt log: it is kept whether prompt logging is on or off, and a zero-retention demand does not remove it. You cannot read it on the console. A Responses turn kept for `previous_response_id` follows the same limit.

llm.buzz keeps a stored answer on the server that answered the first call, and a retry can reach another server. When the key belongs to a request llm.buzz accepted and no answer is stored where the retry arrived, the retry answers 409 `idempotency_replay_unavailable`. It does not run the request again, and it is not billed. This happens when:

- the retry reached a different server than the first call did;
- the answer is older than 24 hours, was dropped to make room, or was lost when llm.buzz restarted;
- the first call ended in an error after llm.buzz had accepted it. There is no answer to store, and the key cannot run the request again. A 502, a 503 with no live provider and a provider's own 400 all end a key this way.

### What to send next

| What you got | What to send |
|---|---|
| A 2xx answer | Nothing. Read the answer. |
| No answer: a timeout, or a connection that closed | The same request with the same key. You get the stored answer, or you join the request that is still running. |
| 409 `idempotency_replay_unavailable` after no answer | The same key again after a short backoff, a few times: a retry that arrives where the first call was served gets the stored answer. If it keeps answering 409, look up the first request on the [Logs](https://app.llm.buzz/console/logs) page. A new key runs and bills the request again, so send one only when a duplicate is acceptable. |
| An error answer worth retrying: 429, 500, 502, 503 or 504 | The request with a new key, after `Retry-After` or a backoff. A new key is always safe; after most of these errors the old key answers 409. |
| A 429 `unavailable_route` with `x-gateway-admission-refused: true` | The same request with the same key, after `Retry-After` (5 seconds). The header means llm.buzz refused the request at capacity before it called any provider: the request ended without effects and cost nothing, so its key can run it once. Only this 429 carries the header. A 429 without it may have reached a provider, so send a new key. |
| A 400, 401, 402, 403 or 404 | Fix the cause, then send the request with a new key. The same key still works after a request that did nothing: a field error (`invalid_parameter`, `unsupported_parameter`, `invalid_json`), a 401, a 403 `key_limit_exceeded` or `management_key_not_allowed`, a 404 `model_not_found`, a 402 `insufficient_credits`, or a 402 `weight_exceeds_budget` or `in_flight_budget_exhausted` refused before the first provider call. After `context_length_exceeded`, `no_allowed_providers` or a 402 `insufficient_quota`, the same key answers 409. |
| 409 `idempotency_conflict` | A new key. This key already belongs to a different request. |

The Python example above follows this table: it reuses the key when it got no answer or a 409, and takes a new key after an error answer, which is also safe after the capacity 429.