Get API key
Text generationImages in requests
Text generation

Images in requests

Models that read images take them inside a message. This page covers images on Chat Completions and Messages, and a message can mix text and several images.

View as MarkdownThe page as plain textOpen in ClaudeAsk questions about this pageOpen in ChatGPTAsk questions about this page
On this page
Which models read imagesChat CompletionsMessagesImages in tool resultsPDFs, audio and videoWhen a request is refusedRelated

Which models read images

A model reads images when image is in architecture.input_modalities on its entry in GET https://api.llm.buzz/api/v1/models. Most text models do today, and this lists them. Before a provider serves a model that takes images, an automated check sends it real requests for each capability the model declares, image input included, and a provider that fails stays out of routing: Which providers serve.

curl -s https://api.llm.buzz/api/v1/models \
  | jq -r '.data[] | select(.architecture.input_modalities | index("image")) | .id'

The vision and image pipelines guide lists the models that read images and the ones that make them.

Chat Completions

An image is an image_url part beside the text of a user message. This request sends a local file as a data: URL:

import base64
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.llm.buzz/api/v1",
    api_key=os.environ["LLM_BUZZ_KEY"],
)
with open("receipt.png", "rb") as image:
    encoded = base64.b64encode(image.read()).decode()
reply = client.chat.completions.create(
    model="deepseek/deepseek-v4.1-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is the total on this receipt?"},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{encoded}"}},
            ],
        }
    ],
)
print(reply.choices[0].message.content)

To send a URL instead, put it in url: {"type": "image_url", "image_url": {"url": "https://example.com/receipt.png", "detail": "high"}}.

  • url is a data: URL with a base64 PNG, JPEG, GIF or WebP image, or an http or https URL. llm.buzz does not download a URL: the provider fetches it. A data URL that is not a base64 image of one of those types is 400 invalid_parameter naming the part.
  • detail is auto, low or high, and is passed on to the provider. Another value is 400 invalid_parameter.
  • media_type is an optional hint, one of image/png, image/jpeg, image/gif and image/webp, for clients such as GitHub Copilot that send it. llm.buzz accepts it and does not send it on: the url and detail decide. A data URL keeps its own type.
  • An image_url object with a key other than url, detail and media_type, or an image_url that is a plain string, is 400 invalid_parameter.
  • Image parts belong in user messages. A tool message may carry them too (Images in tool results), and an assistant message may carry the images a model that makes images returned, sent back to continue the conversation. An image part in a system or developer message is 400 invalid_parameter.
  • An inline image is part of the request body, so it counts toward the 64 MiB limit: a larger body answers 413 request_too_large.

Messages

An image block in a user message takes a source: base64 data with its media_type, or a URL.

import base64

with open("receipt.png", "rb") as image:
    encoded = base64.standard_b64encode(image.read()).decode()
message = client.messages.create(
    model="anthropic/claude-sonnet-5.5",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": encoded}},
                {"type": "text", "text": "What is the total on this receipt?"},
            ],
        }
    ],
)
for block in message.content:
    if block.type == "text":
        print(block.text)

The Anthropic SDK builds client as in Messages. A URL source is {"type": "url", "url": "https://example.com/receipt.png"}. The formats are PNG, JPEG, GIF and WebP. A base64 source of another type, and an image block in an assistant message, is a 400 invalid_request_error naming the field.

Images in tool results

A tool can return an image, such as a screenshot, in the result it sends back to the model. Chat Completions, Responses and Messages each carry it in their own shape, and a provider whose tool results take only text gets the image in the user message that follows: Tools and structured output.

PDFs, audio and video

No chat model on llm.buzz reads PDFs, audio or video today: none lists them in architecture.input_modalities. A model that does lists them there. A Messages document block and a Chat input_audio part reach the same check as an image, so a request that carries one for a model that cannot read it is refused. To turn speech into text, use a transcription model: Audio.

When a request is refused

llm.buzz refuses a part in a user or assistant message that the model cannot read with a 400 before any provider is called, instead of answering from the text around it. That covers an image for a model that does not read images, a URL image when every provider of the model takes inline images only, and a PDF, audio or video part. An image in a tool result is not refused: a provider that takes no image input gets placeholder text where the image was, named messages.content.tool_result.image->placeholder. Chat Completions answers 400 unsupported_capability with metadata.param set to messages. Messages answers 400 invalid_request_error and ends the message with (param: messages). Send the image inline (a data: URL on Chat Completions, a base64 source on Messages) when a URL is refused, or choose a model that lists image in its architecture.input_modalities.

Docs