Skip to main content

Overview

The NanoGPT Batch API runs many independent requests asynchronously at a lower token price than the equivalent synchronous calls. It is a good fit for classification, summarization, evals, synthetic data, document processing, and image analysis where immediate results are not required. Choose a workflow:
  • File-backed API: upload JSONL, create a batch, poll its status, then download output and error files. This OpenAI-compatible workflow is best for large or reusable inputs.
  • Inline API: send the requests in the batch creation body and receive the results when polling the completed batch. This is simpler for smaller jobs.
  • Anthropic Message Batches: send native Claude requests with the official Anthropic SDK or HTTP API. Requests run directly through Anthropic at batch token rates, with native JSONL results.
The file-backed and inline workflows support rows targeting /v1/chat/completions, /v1/responses, or /v1/embeddings. Native Message Batches use a separate /v1/messages/batches endpoint.
All rows in one batch must use the same endpoint and the same model. Batch requests are non-streaming and the only supported completion window is 24h.

Authentication and base URLs

All requests require an API key:
Use the dedicated API host for batch operations: Use api.nano-gpt.com for uploads. The main website host can reject larger multipart requests before they reach the Batch API.

Supported row endpoints

Each row in a file-backed or inline batch must target one of these endpoints:
  • /v1/chat/completions
  • /v1/responses
  • /v1/embeddings
Completions, image generation, audio, video, transcription, TTS, moderation, and other endpoints cannot be used as batch rows. For native Anthropic request bodies, use Message Batches; /v1/messages is not a supported row endpoint for /api/v1/batches.

Model support

Chat Completions batches

/v1/chat/completions batches support OpenAI, Claude, and Gemini models, and open-source models such as openai/gpt-oss-120b, moonshotai/kimi-k3, z-ai/glm-5.2, and deepseek/deepseek-v4-pro. Always submit the canonical NanoGPT model slug; upstream routing is handled automatically. Open-source model rows run through NanoGPT’s normal routing as individual requests, started at about one row per second, so they behave like synchronous requests: they cost the same or less, can choose a provider with the request’s provider field, and accept tools and response_format. They do not accept NanoGPT add-ons such as web search, scraping, memory, characters, or BYOK, and every token limit they set must be greater than 0. Rows for OpenAI, Claude, and Gemini models run on the upstream batch API and do not accept tools, functions, response_format, or structured_outputs. The :thinking suffix is accepted where a thinking variant is configured. Claude thinking aliases are also accepted for compatible models; a numeric thinking budget must be lower than max_tokens. For the live list, call GET /api/v1/models?detailed=true and select models whose supported_batch_endpoints include /v1/chat/completions. Model availability changes. Validate a small batch before submitting a large job; an unsupported model returns an unsupported_model error.

Responses batches

/v1/responses batches currently support direct OpenAI models only. The openai/ prefix is accepted. gpt-5.2-pro and gpt-5.4-pro, including dated snapshots, are not available through the upstream Responses Batch API and are rejected. Responses rows support function and custom tools, structured text output, and remote or data-URL image inputs. Provider-hosted tools and stateful Responses features are not supported in batches.

Embeddings batches

/v1/embeddings batches support text-embedding-3-small, text-embedding-3-large, and text-embedding-ada-002. The openai/ prefix is accepted. Other embedding models return an unsupported_model error that names the supported models. For the live list, call GET /api/v1/embedding-models and select models whose supported_batch_endpoints include /v1/embeddings. Embedding batches with content moderation accept at most 10,000 rows, though a row can hold several inputs. Embedding inputs are checked by the same content moderation as other OpenAI-model batches. One flagged line rejects the whole batch with a 400 that names the lines. Zero Data Retention keys and teams, and EU inference keys, cannot create batches. An API key’s model and provider allowlists also apply: a key whose allowlist does not permit these models, or the provider that serves them, gets a 403.

File-backed API

Endpoints

  • POST /files
  • GET /files
  • GET /files/{file_id}
  • GET /files/{file_id}/content
  • DELETE /files/{file_id}
  • POST /batches
  • GET /batches
  • GET /batches/{batch_id}
  • POST /batches/{batch_id}/cancel

JSONL rules

Each non-empty line must be a JSON object with:
  • a unique, non-empty custom_id
  • method: "POST"
  • url set to /v1/chat/completions, /v1/responses, or /v1/embeddings
  • a body object containing the model and endpoint-specific input
All rows must use the same endpoint and model. stream: true is rejected. Chat Completions rows require a non-empty messages array and a positive max_tokens or max_completion_tokens. They support text and compatible image_url content. Image URLs may use HTTP, HTTPS, or base64 data URLs for PNG, JPEG, GIF, and WebP images. For Gemini models, NanoGPT downloads each HTTP or HTTPS image link when the file is uploaded and stores the image in the batch file, so the model never fetches the link itself. A link must reach a public PNG, JPEG, GIF, or WebP image of at most 30 MB; a web page, another file type, or a private address is refused with media_url_not_fetchable and the line number. The image’s own bytes decide its type, so an image served as a generic download works. Images larger than 3,072 pixels on a side are scaled down, and one still over 5 MB is re-encoded as JPEG. One file can link up to 1,000 different images, and all of them must download within about 75 seconds, so send larger sets as several files or as base64 data URLs. Stored images count toward the 200 MiB file limit.

YouTube videos with Gemini

Batches for google/gemini-3.8-flash, google/gemini-3.7-flash, google/gemini-3.5-flash, google/gemini-3-flash-preview, and google/gemini-3.1-pro-preview also accept public YouTube links as video_url parts in user messages:
Links must use HTTPS and point at a public video: watch, youtu.be, Shorts, live, and embed links work. Each request can link up to two videos. Other video sources, start_offset and end_offset, and YouTube links sent as image_url are rejected. Gemini’s batch upstreams can’t read YouTube links, so when any row in a batch has one, every row in that batch runs through NanoGPT’s normal routing, like open-source model rows. These rows cost the same as synchronous requests, without the 50% batch discount. A video’s length isn’t known until the model reads it, so each video row reserves the model’s full input limit of 1,048,576 tokens at creation. The unused part of the reservation is released when the batch settles. Responses rows require a non-empty input and an integer max_output_tokens of at least 16. Embeddings rows require model and input, and take no output cap. input is a string or an array of up to 2,048 strings; token arrays and multimodal input are rejected. Rows may also set dimensions (text-embedding-3-small and text-embedding-3-large only), encoding_format (float or base64), and user. NanoGPT does not forward user. Any other field is rejected. Each input can be at most 8,191 tokens, and each row at most 300,000 tokens. One batch can hold at most 50,000 inputs across all of its rows. Embedding results are much larger than their input, and result files can be at most 200 MiB. NanoGPT estimates the size at upload and rejects a batch that would not fit with an embedding_result_too_large error. At full size, that is about 5,600 inputs per batch with text-embedding-3-small or text-embedding-ada-002 and about 2,800 with text-embedding-3-large. To fit more, use encoding_format: "base64", which makes results about 4.5 times smaller, or request fewer dimensions, or split the job into several batches.

Chat Completions example

Responses example

Embeddings example

Each successful output row holds a standard embeddings response in response.body. Match rows by custom_id, and use each item’s index to match vectors to the strings in that row’s input.

Upload the file

Create the batch

Set endpoint to the same endpoint used by every JSONL row:

Poll, cancel, and download

Batch statuses are validating, in_progress, finalizing, completed, failed, expired, cancelling, and cancelled. A completed batch normally has an output_file_id; row failures may also produce an error_file_id.

List and delete files

Uploaded input files count toward a staged storage allowance of 1 GiB and 100 files per API key until a batch starts from them. An upload that would exceed it returns 429 with batch_file_storage_quota_exceeded and the space in use. Delete input files you no longer need, and output or error files you have downloaded:
The list follows the OpenAI format (object, data, first_id, last_id, has_more), so client.files.list() and client.files.delete() in the OpenAI SDKs work with these routes. Deletion returns {"id": "file_abc123", "object": "file", "deleted": true} and removes the file’s stored content; batches keep the file ID, but the file can no longer be retrieved, downloaded, or used to create a batch. A file used by a batch that is still running or settling returns 409 with file_in_use; retry after the batch finishes, or cancel it first. Files of inline batches and native Message Batches stay with their batch and also return 409.

Inline API

Inline batches accept up to 10,000 requests, 20 MiB of normalized input, and 250,000 aggregate requested output tokens. Embeddings batches have no output tokens; their estimated results must instead fit in 50 MiB, or about 1,400 inputs at full size with text-embedding-3-small, and more with encoding_format: "base64" or fewer dimensions. Use the file-backed API for larger inputs.

Endpoints

  • POST /batches
  • GET /batches
  • GET /batches/{batch_id}
  • POST /batches/{batch_id}/cancel
All inline routes use the https://api.nano-gpt.com/api/beta base URL.

Create an inline batch

Creation returns 202 Accepted. Poll GET /api/beta/batches/{batch_id} and read results after completion. Cancel an active job with POST /api/beta/batches/{batch_id}/cancel. The batch-level model is inherited by every request body. If an inline row omits its endpoint-specific output cap, NanoGPT applies a 4096-token default. File-backed rows must always include the output cap explicitly. Embeddings rows have no output cap, so no default is applied.

List batches

Use the authenticated list route for the workflow you want to inspect. Each route returns only batches visible to the supplied API key, ordered newest first. Native Message Batches have their own list route and cursor names. Both routes support the same filters and bidirectional cursor pagination: after and before cannot be combined. Cursors are batch IDs, not arbitrary tokens. Creation boundaries accept Unix timestamps in seconds or timezone-qualified ISO 8601/RFC 3339 timestamps with no more than millisecond precision. Supported status values are validating, in_progress, finalizing, completed, failed, expired, cancelling, and cancelled.

Filter inline batches

You can also filter by a strict creation-time range:

Filter all batches

Response and pagination

Both endpoints return the standard list envelope:
  • data is always ordered newest first.
  • first_id and last_id are null for an empty page.
  • To fetch older batches, send after=<last_id> with the same filters.
  • To return to the adjacent newer page, send before=<first_id> with the same filters.
  • has_more indicates that more records exist in the direction being paged.
For example:
Inline list responses contain metadata only. results is always null, including for completed batches. Retrieve GET /api/beta/batches/{batch_id} to obtain a completed inline batch’s results.
Inline list items include id, object, endpoint, model, completion_window, status, created_at, finalized_at, request_counts, usage, error, and a results field that is always null. The OpenAI-compatible list includes both inline and file-backed jobs. Its items retain the standard batch shape, including input_file_id, output_file_id, error_file_id, lifecycle timestamps, request_counts, metadata, usage, and errors where applicable.

Listing errors

Both list routes return:
  • 401 for missing or invalid authentication.
  • 400 when after and before are both present.
  • 400 for an unknown or inaccessible cursor. The inline-only route also rejects a cursor that identifies a non-inline batch. A cursor does not need to match the active filters; it only supplies the page boundary.
  • 400 for an unsupported status value.
  • 400 for an invalid creation timestamp or a reversed creation-time range.

Anthropic Message Batches

Use native Anthropic request shapes for asynchronous Claude jobs, including custom tools, thinking, prompt caching, images, and documents where the selected model supports them. Submit up to 100,000 requests for one supported Claude model per batch. Requests go directly to Anthropic and use batch token pricing, paid from your NanoGPT balance. The official Anthropic SDK works with a NanoGPT API key and baseURL: 'https://api.nano-gpt.com/api' (Python: base_url). It appends /v1/messages/batches. The shorter /v1/messages/batches alias also works with https://api.nano-gpt.com as the SDK base URL.

Native endpoints

All routes also accept the /v1/messages/batches prefix. Authenticate with x-api-key or Authorization: Bearer using your NanoGPT API key.

Create a native batch

Send a requests array containing custom_id and native Messages params. No file upload is required.
Or use @anthropic-ai/sdk:
Choose a new Idempotency-Key for each distinct batch. Reuse the same key and body when retrying a submission; a different payload with the same key returns 409. A stored creation failure can also be replayed, so use a new key for a new submission after correcting that failure. The header is optional, but recommended for retries.

Native request limits and compatibility

  • Send 1–100,000 requests with unique, non-empty custom_id values.
  • Use the same supported Claude model for every request. Thinking controls may vary by request. Support for a model on synchronous /v1/messages does not imply native batch support.
  • Haiku 5.5 accepts anthropic/claude-haiku-5.5, claude-haiku-5-5, and anthropic/claude-haiku-latest. Use thinking: { "type": "adaptive" } with output_config: { "effort": "medium" } to set its default thinking behavior explicitly; fixed budget_tokens controls are unsupported by Haiku 5.5.
  • Haiku 5.5 also accepts thinking: { "type": "disabled" } at low, medium, or high effort. Disabled thinking with xhigh or max is rejected.
  • Each params requires model, messages, and a positive integer max_tokens within the model’s output limit. NanoGPT’s maximum accepted cap is 1,000,000; model limits can be lower. The first message must have the user role; use the top-level system field for a system prompt.
  • The request body and stored normalized input must each fit within 200 MiB. Normalization can make the stored input larger than the original body.
  • The completion window is 24 hours. Streaming generation is unsupported; result downloads use JSONL.
  • Native custom tools and tool-result history, thinking, output_config, and prompt-cache controls are supported where the selected model supports them. Use native thinking controls, not OpenAI reasoning or reasoning_effort fields.
  • Images accept URL or base64 sources; documents accept URL, base64, or plaintext sources. Provider-managed file references, containers, MCP servers, hosted tools, and video blocks are rejected.
  • URL or base64 documents reserve enough balance for a full context window because their token count is unknown before processing. Final billing still uses actual token usage.
  • Caller anthropic-beta flags are forwarded alongside any flags required by the request. A beta header does not enable fields rejected by NanoGPT’s batch validation.
  • Keys or teams requiring Zero Data Retention cannot create batches because batch processing retains input.
Native Message Batches use the official Anthropic SDK. NanoGPT’s general-purpose SDKs do not expose these batch methods.

Native status, results, cancellation, and deletion

Creation returns a message_batch object. Poll retrieval until processing_status is ended; the other states are in_progress and canceling. An ended batch can contain unsuccessful requests, so inspect request_counts and each result:
  • request_counts contains processing, succeeded, errored, canceled, and expired.
  • results_url is available when results can be downloaded. Results are native JSONL rows with custom_id and result; match by custom_id and inspect result.type, rather than assuming input order or success.
  • Results requested before they are available return 409. Download available results before provider archival; an ended status alone does not guarantee results remain available.
  • Cancellation is best-effort. Requests that complete can still incur usage charges.
  • Delete only after processing ends. Deletion hides the batch from NanoGPT’s native API; it does not reverse billing or promise deletion of Anthropic’s retained data.

List native Message Batches

GET /api/v1/messages/batches returns native batches visible to the supplied API key, newest first, with data, has_more, first_id, and last_id. Use limit from 1 to 100 (default 20), after_id=<last_id> for older batches, or before_id=<first_id> for newer batches. Do not combine both cursor parameters. Use the native list for native Message Batches; the /api/v1/batches list uses the OpenAI-compatible representation.

Responses Batch restrictions

Responses Batch is stateless and executes directly through the upstream batch service. NanoGPT forces store: false and rejects:
  • previous_response_id, conversation, and background: true
  • reusable prompt references
  • input_file, item_reference, video inputs, and input_image.file_id
  • provider-hosted tools; function and custom tools remain supported
  • NanoGPT-only features such as Advisor, memory, scraping, retention overrides, provider or BYOK controls, caching controls, and billing overrides
Remote HTTP(S) image URLs and image data URLs are accepted. Structured output through the Responses text.format field is supported.

Billing

Batch jobs use NanoGPT account balance and do not use subscription included tokens. At creation, NanoGPT checks the balance against a conservative maximum-liability estimate based on the input and output caps. Completed usage is charged once after the batch reaches a terminal state. If there is no billable usage, no usage charge is created.
  • OpenAI, Claude, and Gemini models, and some fine-tuned models: token usage is priced 50% below the equivalent synchronous request. Non-token charges, where supported, keep their normal rate. A Gemini batch with a YouTube video is the exception: its rows are charged like open-source model rows.
  • Other open-source models: each row is charged exactly what the same synchronous request costs, including the automatic routing price, your subscriber discount, and the price of any provider you select. Rows are priced in USD, and a batch never costs more than the amount reserved when it was created.
  • Open-source models with a batch price: some models, such as moonshotai/kimi-k3, have a lower batch price for rows that use automatic routing. An alias such as moonshotai/kimi-latest uses the batch price of the model it points to. A row that sets provider pays that provider’s normal price instead. A row never costs more than the same synchronous request.
  • Embedding models: input tokens are priced 50% below the synchronous embedding price: $0.01, $0.065, and $0.05 per million tokens for `text-embedding-3-small`, `text-embedding-3-large`, and `text-embedding-ada-002`. There are no output tokens. At creation, NanoGPT holds the estimated input cost plus 20%, with a $0.0001 minimum, then charges the tokens reported for successful rows. Failed rows are not charged.
Each model page shows its Batch API price. Use the live pricing page or pricing API as the source of truth.

Common errors

  • Unsupported endpoint: file-backed and inline rows must use /v1/chat/completions, /v1/responses, or /v1/embeddings, consistently across all rows. Native Claude requests use /v1/messages/batches.
  • Missing output cap: add max_tokens or max_completion_tokens for Chat Completions, or max_output_tokens >= 16 for Responses.
  • Mixed model: every row must use the same model.
  • Streaming unsupported: remove stream: true.
  • Unsupported Responses model: choose a direct OpenAI model supported by the upstream Batch API.
  • Unsupported Responses field: remove stateful features, provider-hosted tools, file references, or NanoGPT-only extensions.
  • Unsupported embeddings input: input must be a string or an array of strings. Token arrays and multimodal input return unsupported_input_format.
  • Embedding results too large: embedding_result_too_large (file-backed) or inline_result_too_large (inline). Use encoding_format: "base64", fewer dimensions, or smaller batches.