Retries and idempotency

Inside one request

A model usually has several providers. Until a provider sends its first text, tool call or refusal, a provider that fails or times out is replaced by the next one, in the order the model's routing preset gives (How a provider is chosen). llm.buzz does not call a provider again for an ordinary failure. The answer's x-gateway-provider names the provider that answered, and x-gateway-route-depth says where it stood in the order, counting from 0.

Replacing a provider stops when:

• Output has started. The first text, tool call or refusal fixes the provider for the rest of the request, and llm.buzz never splices another provider's output into it. A later failure ends the answer with an error, or as incomplete when the provider's stream just stops: Streaming.
• The model refused. A refusal reaches you from the provider that refused it and is not tried on another.
• The provider rejected the request. A provider's own 400, whose message starts provider rejected the request:, reaches you and is not tried on another provider.
• A ceiling is reached. A request makes at most 8 provider calls and runs for at most 600 seconds, and the bounds you set with gateway apply.

What a Chat Completions, Responses or Messages request costs when providers fail:

Embeddings, Images and Audio bill a failed last attempt by their own rules, often nothing: Embeddings, Images and Audio.

A request llm.buzz refuses before it calls any provider costs nothing. That covers a 400 for a bad field, a 401, a 404, a 402 for a spent balance, and a 503 when no provider is live.

The gateway object

gateway sets bounds for one request on Chat Completions, Responses and Messages:

• retry.max_attempts_per_route caps the calls to one provider, 1 to 4. It is a cap and adds no retries, because a failed provider is not called again.
• retry.max_total_attempts caps the calls in all, 1 to 8, fallbacks included. It is 8 when you leave it out.
• retry.backoff sets the wait before a throttled provider is called again.
• routing.allow_fallbacks: false keeps the request on the first provider, as provider.allow_fallbacks: false does.

These are ceilings: your spending limits, the 600-second limit and a provider's health can stop a request sooner. Retries and fallbacks per request has the fields and an example. A retry with an Idempotency-Key must send the same gateway object: a changed one is a 409 idempotency_conflict.

Your retries

Your own retries are requests of their own. Every one that runs is billed, and an error answer may have cost something, as the table above says.

The OpenAI and Anthropic SDKs retry on their own, twice by default, after a connection error and after a 408, 409, 429 or 5xx answer. Each of their retries is a request of its own, and without an Idempotency-Key it runs and bills again. If you keep the SDK's retries and send an Idempotency-Key, the SDK resends the same key. After a dropped connection that replays safely, and after a 429 with x-gateway-admission-refused: true the retry runs the request once. After a 502, a 503 or a 429 unavailable_route without that header, its retries end in 409 idempotency_replay_unavailable, which is then the error you see. Set max_retries=0 in Python or maxRetries: 0 in TypeScript when you handle retries yourself, or when one operation must bound the work it does. Which Idempotency-Key to send with a retry is covered below.

Idempotency-Key

Send an Idempotency-Key header and a retry with the same key and the same body never runs the request twice. Use one key for each distinct request, such as a UUID, and send the same key only when you retry that request.

import os
import time
import uuid

from openai import APIConnectionError, APIStatusError, ConflictError, OpenAI

client = OpenAI(
    base_url="https://api.llm.buzz/api/v1",
    api_key=os.environ["LLM_BUZZ_KEY"],
    max_retries=0,
)


def summarize(text: str) -> str | None:
    key = str(uuid.uuid4())  # one key for this request and its retries
    for attempt in range(5):
        try:
            reply = client.chat.completions.create(
                model="deepseek/deepseek-v4.1-flash",
                messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}],
                max_tokens=200,
                extra_headers={"Idempotency-Key": key},
            )
            return reply.choices[0].message.content
        except (APIConnectionError, ConflictError):
            # No answer, or 409 idempotency_replay_unavailable: the request may have run.
            # Ask again with the same key.
            delay = 2**attempt
        except APIStatusError as error:
            if error.status_code not in (429, 500, 502, 503, 504):
                raise
            # An error answer ended the request, and its key cannot run it again.
            key = str(uuid.uuid4())
            delay = float(error.response.headers.get("retry-after", 2**attempt))
        if attempt < 4:
            time.sleep(delay)
    raise RuntimeError("no answer after 5 attempts")


print(summarize("HTTP clients retry with backoff so they do not add to an outage."))

import OpenAI from "openai";
import { randomUUID } from "node:crypto";

const client = new OpenAI({
  baseURL: "https://api.llm.buzz/api/v1",
  apiKey: process.env.LLM_BUZZ_KEY,
  maxRetries: 0,
});
const key = randomUUID(); // one key for this request and its retries
const reply = await client.chat.completions.create(
  {
    model: "deepseek/deepseek-v4.1-flash",
    messages: [{ role: "user", content: "Summarize in one sentence: HTTP clients retry with backoff." }],
    max_tokens: 200,
  },
  { headers: { "Idempotency-Key": key } },
);
console.log(reply.choices[0].message.content);

KEY=$(uuidgen)
for attempt in 1 2; do
  curl -s https://api.llm.buzz/api/v1/chat/completions \
    -H "Authorization: Bearer $LLM_BUZZ_KEY" \
    -H "Idempotency-Key: $KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "deepseek/deepseek-v4.1-flash",
      "max_tokens": 200,
      "messages": [{"role": "user", "content": "Summarize in one sentence: HTTP clients retry with backoff."}]
    }' | jq -r '.id, .usage.cost'
done

The curl loop prints the same id and cost twice: the second call is the stored answer of the first.

What a retry gets:

• After the first call answered: the stored answer, with its original cost and x-request-id. No provider is called and nothing is billed. A stream comes back as the same stream.
• While the first call is still running: the retry waits for it and returns its answer. It does not start a second generation.
• After you closed a keyed stream: the request keeps running and is billed when it finishes, and a retry with the key joins it or gets its answer. A Chat Completions, Responses or Messages stream without a key stops reading from the provider when you close it.
• With a different request: 409 idempotency_conflict. Any change to the body counts, including stream, the gateway object, a provider hint and a model suffix such as :floor. A changed stream_options.include_usage does not count. Send a new key for each distinct request.
• Under another data-policy demand: a key replays only for a call that makes the same data-policy demand as the first one did: no demand, zero retention, or no training and no retention, whether it came from provider.zdr, provider.data_collection or your organization's data policy. The call must also run under the same residency, EU or anywhere. A retry that makes another demand is not a replay: it runs and bills as a new request.
• After a limit or a spent balance: a retry of a request that completed still returns its stored answer once the key has reached its spending limit or your balance is spent. A new request is refused.

x-client-request-id, session_id and x-session-id never act as a key. Two requests with the same prompt and no Idempotency-Key are two requests.

When the answer is not stored

llm.buzz keeps the answer to a keyed request for up to 24 hours, or less when it needs the room. This copy is not a prompt log: it is kept whether prompt logging is on or off, and a zero-retention demand does not remove it. You cannot read it on the console. A Responses turn kept for previous_response_id follows the same limit.

llm.buzz keeps a stored answer on the server that answered the first call, and a retry can reach another server. When the key belongs to a request llm.buzz accepted and no answer is stored where the retry arrived, the retry answers 409 idempotency_replay_unavailable. It does not run the request again, and it is not billed. This happens when:

• the retry reached a different server than the first call did;
• the answer is older than 24 hours, was dropped to make room, or was lost when llm.buzz restarted;
• the first call ended in an error after llm.buzz had accepted it. There is no answer to store, and the key cannot run the request again. A 502, a 503 with no live provider and a provider's own 400 all end a key this way.

What to send next

The Python example above follows this table: it reuses the key when it got no answer or a 409, and takes a new key after an error answer, which is also safe after the capacity 429.