Text generation

Streaming

With stream: true the answer arrives as server-sent events while the model writes it. Chat Completions, Responses and Messages all stream, each with its own events.

On this page

Stream a request Events in each API Comments Failures Stopping a stream What never streams Related

Stream a request

The SDKs read the events for you. This loop prints each piece of text as it arrives, and the last chunk carries the usage and its cost. You do not need stream_options: the usage chunk comes whenever the counts are known.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.llm.buzz/api/v1",
    api_key=os.environ["LLM_BUZZ_KEY"],
)
stream = client.chat.completions.create(
    model="deepseek/deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Explain a database index in two sentences."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n{chunk.usage.total_tokens} tokens, cost {chunk.usage.model_extra.get('cost')}")

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.llm.buzz/api/v1",
  apiKey: process.env.LLM_BUZZ_KEY,
});
const stream = await client.chat.completions.create({
  model: "deepseek/deepseek-v4.1-flash",
  messages: [{ role: "user", content: "Explain a database index in two sentences." }],
  stream: true,
});
for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content;
  if (text) process.stdout.write(text);
  if (chunk.usage) {
    const { cost } = chunk.usage as { cost?: number };
    console.log(`\n${chunk.usage.total_tokens} tokens, cost ${cost}`);
  }
}

curl -N https://api.llm.buzz/api/v1/chat/completions \
  -H "Authorization: Bearer $LLM_BUZZ_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v4.1-flash",
    "stream": true,
    "messages": [{"role": "user", "content": "Explain a database index in two sentences."}]
  }'

-N turns off curl's own buffering, so the lines print as they arrive. Responses and Messages take the same stream: true and send their own events, described next.

Events in each API

	Chat Completions	Responses	Messages
Opens with	A chunk with the assistant role	`response.created`, then `response.in_progress`	`message_start`, with the message so far
Content	`data:` lines, each a `chat.completion.chunk` with `delta.content`, `delta.reasoning`, `delta.tool_calls` or `delta.images`. Every chunk carries `provider`.	OpenAI's output events for text, function-call arguments and reasoning summaries, such as `response.output_text.delta`	`content_block_start`, `content_block_delta` and `content_block_stop` for text, tool input and thinking. One `ping` follows `message_start`.
Stop reason	`finish_reason` on the finish chunk, whose `delta` is empty; the usage chunk repeats it	`status` of the final response	`stop_reason` on `message_delta`
Usage and cost	A last chunk with `usage`, `cost` inside it	`usage`, `cost` inside it, on the final response	`usage`, `cost` inside it, on `message_delta`. `message_start` carries an early `usage` and never `cost`.
Ignored fields	`x-sgr-ignored-parameters` on content and finish chunks, not on the usage or error chunk	On the response object in the events	On `message_start`
Ends with	`data: [DONE]`	Exactly one terminal event, which carries the final response	`message_stop`
A failure after output	One chunk with an `error` object and `finish_reason: "error"`, then `data: [DONE]`	The terminal event reports the failure	One `error` event in place of `message_stop`

x-sgr-ignored-parameters is a key in the body of an event, not a header. cost is left out of usage when any part of the cost is unknown. The usage in message_start is an early figure: Anthropic's own start counts when Anthropic serves the request, and llm.buzz's estimate of the prompt for any other provider. message_delta carries the final counts: Usage and cost. The response object in Responses events also echoes instructions, metadata and tools: The answer.

Comments

While the model is silent, llm.buzz sends comment lines, which begin with :, to keep the connection active. Any server-sent events client ignores them; ignore them in a parser of your own too. They carry no data, and they do not extend the 600-second limit below.

Failures

Before output reaches you, a provider that fails or times out is replaced by the next provider of the model, unless you turned fallbacks off, and the failed attempt costs nothing; a provider's own 400 is not tried on another provider. Your stream opens from the provider that answered. When every provider fails before it opens, you get an ordinary HTTP error: Errors.
Once output has reached you, the request stays on the provider that sent it. The first text, tool call or refusal that reaches you fixes the provider, and llm.buzz never splices another provider's output into the stream. A refusal is not retried on another provider.
A later failure ends the stream with an error in your API's form, shown in the table above. The HTTP status was already 200 when the stream opened, so read the last event of every stream before you treat the answer as complete.
A provider that stops after output, closing its stream without a terminal event, is not an error. llm.buzz finishes the answer from what was served: with the finish the provider had already declared, when it answers in OpenAI's Chat Completions form, or otherwise as incomplete (finish_reason: "length" on Chat Completions, status: "incomplete" on Responses, stop_reason: "max_tokens" on Messages), with open items closed and a tool call cut in the middle dropped. A provider that stops before any output is replaced by the next provider.
At 600 seconds a stream still generating ends with a timeout error, followed on Chat Completions by data: [DONE], and is billed for the prompt and the output you received. The request time limit is shared by every provider the request tried.
A balance that runs out while a stream runs does not cut it off. llm.buzz charges the stream when it completes.

Stopping a stream

On Chat Completions, Responses and Messages, closing the connection makes llm.buzz stop reading from the provider, provided the request carried no Idempotency-Key. Messages has no such key, so closing a Messages stream always stops it. A provider may still finish work it had begun.

You are billed for the prompt and for the output you received. When the provider has not reported the counts, llm.buzz counts them itself.
A stream you close before the provider accepted the request is charged what llm.buzz reserved for it before calling the provider, the most it could cost.
A Chat Completions or Responses stream sent with an Idempotency-Key keeps running after you leave, within the limit of what llm.buzz keeps for a replay, and a retry with the same key joins it or returns its answer instead of starting a second generation. The request is billed once: Retries and idempotency.

What never streams

Embeddings and Images answer 400 to stream: true, Audio returns its whole result at once, and Decisions are buffered and refuse a stream field. Responses also has a WebSocket transport that carries the same events, one JSON event in each frame: WebSocket.

Chat Completions, Responses and Messages list what each API adds to a stream.
Tools and structured output covers streamed tool arguments and what a cut-off call looks like.