Batch jobs

Some providers run a batch lane: you upload a file of requests and collect the answers later. llm.buzz does not run one today. Its /batches and /files routes answer 404, “batches are not enabled on this gateway”. A large job on llm.buzz is many single requests, and this guide is how to send them well.

Pick a cheap model

A bulk job multiplies every price by the number of requests. These are the cheapest text models live right now, by input price and then output price, in US dollars per million tokens:

And 40 more models on Models.

From the public models list, read 27 October 2026, 05:52 UTC.

Each price is the model’s first-ranked provider’s, with any running discount applied. Add :floor to the model’s name, openai/gpt-5-nano:floor, and the cheapest provider of that model goes first on every request.

Give every request a key

On Chat Completions and Responses, send an Idempotency-Key header with a value unique to the item, such as its row ID. A retry with the same key and the same body never runs the request twice, so a retry after a timeout on your side is never billed twice: it returns the first answer, or 409 idempotency_replay_unavailable when that answer is not stored where the retry arrived.

reply = client.chat.completions.create(
    model="openai/gpt-5-nano:floor",
    messages=[{"role": "user", "content": f"Classify this ticket: {ticket.text}"}],
    extra_headers={"Idempotency-Key": f"tickets-{ticket.id}"},
)

Use a new key for each distinct request: reusing one with a different body answers 409 idempotency_conflict.

On idempotency_replay_unavailable, llm.buzz accepted the item and did not run it again. It may still be running, or its answer may exist, so retry it with the same key after a short backoff first: the retry may reach the stored answer. Sending it without the key, or with a new one, runs it again and bills it again, so do that only when a duplicate is acceptable, or when the first call ended in an error answer, which leaves its key answering 409 for good. What to send next has the cases.

Retry what can be retried

Retry a 429, 502, 503 or 504 with backoff, and wait Retry-After seconds first when the answer carries it. After an error answer, send the retry with a new key: the old key answers 409 idempotency_replay_unavailable after most errors (What to send next). A 429 unavailable_route means providers are throttling or the model is at capacity: slow down rather than give up. Do not resend any other 4xx unchanged. Fix the request, the key or the balance first.

Watch your balance

Every request in flight holds part of your balance until it settles. Two answers tell you a job is pressing on it:

Setting max_tokens to what an item needs also keeps each request’s hold small. Every answer carries usage.cost, the request’s cost in US dollars when every part of it is known, so your job can add up what it spent as it goes.

Share a prefix

Bulk requests often start with the same long instructions. Most providers cache a repeated prefix on their own and bill it at their cached-input price. Claude models cache only where the request marks it: send Anthropic’s cache marker, cache_control: {"type": "ephemeral"}, which llm.buzz passes on and never adds itself. Cached input shows in usage.prompt_tokens_details.cached_tokens on Chat Completions. Prompt caching has the details.

Mind the time limit

A request may run for 600 seconds. Past that, it answers 504 deadline_exceeded. When an item hits it, retry, or ask for a shorter answer.