Provider Selection
Provider selection chooses the upstream provider for a supported model. It does not change the model ID. For example, to use Kimi K2.6 through Novita, keep the model asmoonshotai/kimi-k2.6, or another supported alias for that model, and send the provider separately with X-Provider.
Provider selection is optional; if you do nothing, the platform picks the provider. X-Provider, a body provider string, or a body provider object can select or constrain providers for a request. Explicit provider selection is always pay-as-you-go and is charged at the selected provider’s price, including provider-selection markup. For subscription users, sending an explicit provider selection bypasses subscription coverage for that request.
X-Billing-Mode: paygo is only needed when forcing pay-as-you-go billing without an explicit provider, or when saved provider preferences should apply to subscription-included traffic. See Pay-As-You-Go Billing Override.
This page intentionally documents only public provider IDs returned by the provider-discovery endpoints. Internal routing/provider names are never part of the public API contract.
Choose Your Routing Goal
See examples and precedence before combining these controls. Cache contents are provider-local; changing providers does not transfer a warm cache.
When Provider Selection Applies
- Provider selection only applies when a model reports
supportsProviderSelection: true. - Use
GET /api/models/:canonicalId/providersto discover eligible models and provider IDs. supportsProviderSelectionis exposed on the model-specific provider endpoint. Do not rely on/api/v1/modelsor/api/v1/models?detailed=trueto determine provider-selection support.- If a model does not support provider selection, it runs on a fixed route, and ranking preferences such as a soft
orderorsorthave no effect. Hard limits still apply:only,ignoreandorderwithallow_fallbacks: false. If the route can’t meet them, the request returnsno_allowed_provider_for_model. - You cannot currently force a provider and have that same request count as subscription-included usage.
Google Vertex and Google AI Studio
Gemini models can be served by two separate Google services, and each is its own provider:googlemeans Google Vertex inprovider.only,provider.orderandX-Provider. Nobody is moved to AI Studio without choosing it.provider.ignore: ["google"]removes both services. Usegoogle-vertexorgoogle-ai-studioto remove just one.- Vertex no longer serves the Gemini 2.5 text models. A request that allows only Vertex for one of them gets an error naming Google AI Studio instead of being moved there.
Discover Providers and Pricing
Use this endpoint to list available providers and the pricing you will pay when selecting one.distillationPolicy, which indicates whether that specific hosted provider route is allowed for output-based model training or distillation under NanoGPT’s recorded provider-terms and model-license rules. See Distillation Policy.
This endpoint is not under /api/v1. If your base URL is https://api.nano-gpt.com/api/v1, do not append this path to that base URL. Use:
/ in the model ID as %2F, as shown above. The unencoded slash is treated as a path separator:
:canonicalId means the model ID or one of its supported aliases.
Example response:
defaultPriceis used when you do not select a provider.providers[].pricingincludes provider-selection markup. Fields ending inPer1kTokensare USD per 1,000 tokens; multiply by 1,000 to compare withmax_price, which is USD per 1 million tokens.- When present,
providers[].effectivePricing.effective_pricedescribes the effective rates used for routing, including current discounts and markup. Prices and availability can change between discovery and the request. - Use the exact value from
providers[].providerin theX-Providerheader. - Provider-specific
distillationPolicycan differ from the model-level policy. Explicit provider restrictions override model-level allowances. - If a model is unsupported, the response includes
supportsProviderSelection: false.
Provider row fields
Optional fields are omitted when NanoGPT has no reliable value for that provider.Per-Request Provider Override
For a single request, send the provider ID in theX-Provider header.
X-Provideris case-insensitive (x-provideralso works).- The provider ID must be one of the values returned by the providers endpoint.
- The body
providerfield also accepts the same provider ID as a string onPOST /api/v1/chat/completions,POST /api/v1/completions, andPOST /api/talk-to-gpt. - Recommended: use
X-Provideror bodyproviderfor explicit provider overrides. The API also accepts a trailing provider suffix for user-selectable providers, such asmodel-id:cerebras, but request fields are clearer and easier to validate against provider-discovery responses. - To avoid Moonshot for Kimi K2.6, select any available non-Moonshot provider ID returned by the provider-discovery endpoint, such as
novita,cloudflare,baseten, orinceptron. - If the request would otherwise be subscription-covered, explicit provider selection still bypasses subscription coverage and bills the request as pay-as-you-go. You do not need to also send
billing_mode: "paygo"orX-Billing-Mode: paygo. - If Enable paid models on API is off and the request doesn’t force pay-as-you-go with
billing_mode: "paygo"orX-Billing-Mode: paygo, a provider pick returns an error and nothing is charged.X-Provideror aproviderstring returns403 provider_selected.provider.only,provider.ignoreorprovider.allow_fallbacks: falsereturns400 provider_selection_not_available. Otherproviderobject fields, such asorderandsort, are ignored for that subscription.
Provider Routing Object
POST /api/v1/chat/completions, POST /api/v1/responses, POST /api/v1/messages, POST /api/v1/completions, and POST /api/talk-to-gpt also accept a structured provider object for per-request routing controls. See Provider controls on Responses and Messages for how those two routes apply them.
GET /api/models/:canonicalId/providers to discover public provider IDs for a model. Do not rely on internal provider names that are not returned by public provider-discovery responses.
Examples:
Routing Semantics
orderis soft unlessallow_fallbacksisfalse, which makes it a hard list. Useonly, ororderwithallow_fallbacks: false, when the request must fail instead of using any provider outside the list.onlyis hard. It restricts routing to the listed providers and disables internal fallback outside the pinned set.ignoreandmax_pricecan resolve to a concrete provider. For direct provider routing, NanoGPT cannot represent “any provider except X” as a multi-provider pool. If filters likeignoreormax_priceare provided withoutorderoronly, NanoGPT resolves to one cheapest concrete provider that satisfies those filters. That is treated as explicit provider selection for billing and routing because the request changed the provider candidate set.- If
max_priceis provided withoutorderor an explicitsort, NanoGPT treats it as a request for price-aware routing among providers that satisfy the cap.onlydoes not prevent this inference; anorderdoes. - The request’s output budget (
max_tokens,max_completion_tokensormax_output_tokens) is a hard routing constraint too; see output limits. - Quantization filters are hard routing constraints. If no available provider satisfies the requested quantization, provider list, price, fallback, caching, and parameter constraints, the request may fail with a provider-availability error.
- Per-request controls can change routing order, but saved hard restrictions still apply. Saved provider allowlists are intersected with request allowlists, exclusions remain excluded, and a saved fallback restriction is preserved. A request cannot broaden a saved allowlist by sending
order,only, orallow_fallbacks: true.
Quantization Filters
Use quantization filters when you need to avoid lower-precision provider routes for a model. Choose a model with provider selection and published quantization metadata, such asdeepseek/deepseek-v4.1-flash-1017 in the examples below.
provider.quantizations is an exact allowlist. It must be a non-empty array:
unknown explicitly:
provider.min_quantization sets a minimum precision floor:
min_quantization: "fp8" allows int8, fp8, fp16, bf16, and fp32. Integer and floating-point formats with the same bit width are treated as equivalent for minimum filtering.
You may combine exact and minimum filters. NanoGPT uses the overlap:
quantizations: ["fp4"] with min_quantization: "fp8" is invalid because fp4 is below the requested minimum.
Use NanoGPT’s canonical quantization values in API requests. Provider metadata may use different internal labels, but request parameters must use the supported canonical values listed above.
Performance cutoffs
preferred_max_latency and preferred_min_throughput move providers that miss a cutoff to the end of the ranking. They never exclude a provider and never make a request fail: if no provider meets the cutoffs, or NanoGPT has no recent data, the ranking stays as it was.
- What a percentile means.
p90: 3on latency means 90% of recent requests got their first token within 3 seconds.p90: 50on throughput means 90% of recent requests ran at 50 tokens per second or faster. Setting several cutoffs requires all of them. - They need a ranking to reorder. Cutoffs apply when
provider.sort, or aprovider.max_pricethat implies price routing, ranks several providers in the sameproviderobject. A provider list or filter that resolves to a single provider has nothing to reorder. - On their own, they do nothing. Sent without any other control that chooses among providers, cutoffs leave automatic routing and your bill as they were. Add
provider.sortto use them, which bills the request as explicit provider selection. - They can’t widen a restriction. They only reorder the providers your request, API key, team policy, zero data retention and EU inference settings already allow.
- How the numbers are measured. They cover about the last hour of requests and are estimated from fixed ranges, not exact samples. A provider needs at least 10 recent requests on that model to count as meeting a cutoff, so one with less traffic is treated as missing it.
POST /api/v1/completions and POST /api/talk-to-gpt, cutoffs are accepted and have no effect.
Model fallbacks
Sendmodels to list other models to try if one fails. It works on POST /api/v1/chat/completions, POST /api/v1/responses and POST /api/v1/messages. A list that comes down to one model, once duplicates are removed, is a normal request for that model. Everything below about trying several models applies to two or more different models.
- Order.
modelgoes first, thenmodels. You can leave outmodeland send onlymodels. Duplicates are removed.modelscan list at most 4 IDs, and a request can try at most 4 models in total. - What moves to the next model. A provider failure (a 5xx, or a provider’s 429) and a prompt too long for that model, once NanoGPT’s own routing for that model has run out of options.
- What doesn’t. Authentication, permission, balance and invalid-request errors, moderation refusals, and NanoGPT’s own rate limits, concurrency limits and budgets. The request ends with that error.
- Streaming. Fallback happens only before a response starts. Once NanoGPT returns a
200, including a stream, the request stays on that model, even if the stream later fails. - The final error. If every model fails, you get the last model’s error.
- What you see. The response’s
modelis the model that answered.x_nanogpt_pricing.requestIdidentifies that attempt, and it’s the ID to use for a request billing lookup.X-Request-IDkeeps the first attempt’s ID. Which provider was used isn’t revealed. - Billing. Each attempt is billed as if you had requested that model alone, including whether a subscription covers it. The model that answered is billed at its own price. A failed attempt is free unless its provider charged for it, for example for output generated before the failure or an empty answer the provider billed. Such an attempt is billed under its own request ID, as it would be without
models. - Limits and policies. Each attempt passes the same checks as a request for that model alone, including your API key’s model and provider allowlists, team policy, zero data retention, EU inference and balance.
providersettings apply to every model in the list. A model those rules refuse ends the request with that error. On/v1/responsesand/v1/messages, each attempt also counts toward your rate limits. Attempts run one at a time, and a failed attempt’s concurrency slot is freed before the next one starts. provider.sort.partition. Models are always tried in order, which is whatmodelmeans.none, which would rank across models, isn’t supported withmodelsand returns400provider_sort_partition_not_supported.
400:
A malformed list returns
models_invalid, and more than 4 entries in models, or more than 4 models in total, returns models_too_many. These two apply to any models list, however many models it names. A web_search tool on /v1/responses isn’t the add-on, and can be used with models.
Provider controls on Responses and Messages
Both routes accept the sameprovider object as Chat Completions, with the same errors, and /v1/messages applies it the same way.
Most /v1/responses requests also run through the same routing. Some can only run on a native route, for example those using built-in tools or computer use. A native route doesn’t rank providers, so:
Output limits
Providers can accept different maximum output budgets for the same model. A provider whosemaxOutputTokens is lower than the request’s max_tokens, max_completion_tokens or max_output_tokens is skipped, including under sort: "price" and max_price routing. NanoGPT does not lower your budget to fit a provider.
For example, with max_tokens: 131072, a provider whose maxOutputTokens is 128000 is never chosen, even when it is the cheapest. To keep a provider eligible, send a budget at or below its maxOutputTokens, or omit max_tokens. Check providers[].maxOutputTokens from the discovery endpoint before sending large budgets.
Validation
Invalid object shapes return a structured400 error:
provider.order,provider.only, andprovider.ignoremust be arrays of strings.provider.quantizationsmust be a non-empty array of supported quantization strings.provider.min_quantizationmust be a supported comparable quantization string.unknownis only valid inprovider.quantizations.- If
provider.quantizationsandprovider.min_quantizationhave no overlap, the request is invalid. - Unknown providers in
onlyreturnprovider_unknown_provider. - Unknown providers in
orderandignoreare ignored. The exception isprovider.allow_fallbacks: false, which makesorderthe full list, enforced likeonly. Ifordernames no usable provider, the request returnsprovider_unknown_provider. If it shares none withonly, the request returnsprovider_order_only_conflict. If Enable paid models on API is off for a subscription, it returnsprovider_selection_not_available. provider.data_collectionmust beallowordeny.denyreturnsprovider_data_collection_unsupported, unless the request also sendsprovider.zdr: true. To require zero data retention, sendprovider.zdr: true.provider.max_price.promptandprovider.max_price.completionmust be non-negative numbers.provider.allow_fallbacksandprovider.require_parametersmust be booleans.provider.sortmust be a string or an object withbyand an optionalpartition. Invalid values returnprovider_sort_invalid.provider.preferred_max_latencyandprovider.preferred_min_throughputmust be a non-negative number or an object with non-negativep50,p75,p90orp99values. Invalid values returnprovider_field_invalid.
Consecutive Prompts and Prompt Caching
When you explicitly select a provider withX-Provider or body provider, NanoGPT routes that request to the selected provider. For consecutive prompts, using the same provider helps keep routing stable, which is the setup you want for provider-side prompt-cache reuse when that provider and model support caching.
For capability-based routing, use top-level caching: true or the :caching model suffix instead of choosing a specific provider. NanoGPT will route to any available provider that supports prompt/input caching for the requested provider-selection model, and later matching requests prefer the same eligible provider through soft affinity.
General automatic routing may use cache-affinity routing for eligible cache-capable providers, but it does not require a cache-capable provider and may still choose different upstream providers when request shape, provider availability, or routing constraints change. For cache-sensitive workflows, either consistently send the same X-Provider value or use caching: true / :caching when any cache-capable provider is acceptable.
Cache-Capable Provider Routing
Setcaching: true when you want NanoGPT to route a chat completion request to any available provider that supports prompt/input caching. This is capability-based provider selection, not explicit provider selection and not prompt-cache annotation. It does not add Anthropic-style cache_control markers or configure cache TTLs.
caching: true prefers the provider from your previous matching request (soft affinity). stickyProvider defaults to false; set it to true to disable cross-provider fallback. Neither setting guarantees a cache hit.
To prevent cross-provider fallback when preserving the cache route matters more than availability:
stickyProvider is accepted as a camelCase alias for stickyprovider. Setting it to true disables cross-provider fallback for the caching request. Leaving it false allows fallback; it does not disable the soft affinity established by explicit caching.
Affinity expires after about one hour without a successful matching request. This is a routing preference lifetime, separate from the provider’s cache TTL. Availability, exclusions, price caps, and request compatibility still apply.
The model suffixes :caching, :cache, and :cached request the same cache-capable provider routing:
caching: true:
- Filter to providers that are available, not excluded by preferences, and marked as prompt-caching capable.
- Prefer the previously recorded provider for the same cache-relevant request shape when still usable.
- Try eligible per-model and global provider preferences.
- Otherwise use provider prices and recent aggregate cache-read observations to choose a route. A provider with better observed cache reuse may be preferred within a bounded price range.
Price Caps and Cache Reuse
Use public provider IDs from the discovery endpoint and choose caps suitable for that model. The prices below are illustrative.Lowest estimated cost with caps
One provider with caps
only leaves a single candidate. This request fails if the provider is unavailable, incompatible, or over the cap. Keeping the provider fixed helps cache reuse, but does not guarantee a cache hit.
Cache reuse with automatic provider choice
Combining controls
An ordered provider list is evaluated for each request. It is not a sticky failover session: failure on one request does not permanently advance the list to the next provider. In caching mode, an eligible provider recorded by affinity can take priority over the order.
Price-cap limits
For requests with a verified provider-selection price,max_price restricts retries to routes whose billed base rate NanoGPT can verify against your cap. If the selected provider fails and no such route is available, the request fails instead of being billed at an unverified rate. On models without provider selection, max_price only filters routes; billing uses the model’s price. An explicitly selected gateway without a verifiable provider price rejects the cap.
max_price compares effective provider input/output rates after provider-selection markup, so a provider with a raw input rate of $0.06/M and 5% markup requires a prompt cap of at least $0.063/M. The selection markup is already included in provider-discovery prices; do not add it again.
These are token-rate filters, not a total request budget. They do not separately cap cache-read prices, cache-write prices, search fees, or other add-ons. Local comparisons use the provider’s base input/output tier, so inspect higher-context tiers separately. Unknown prices cannot be verified locally; gateway routes also pass a price constraint upstream, while direct fallback routes with unknown prices are excluded. For a predictable choice, select a provider with published pricing and disable fallback.
Per-Request Routing Preference
For provider-selection-capable models, you can append a routing preference suffix to the model ID::speed/:fast: best estimated completion time using TTFT plus TPS.:throughput: highest tokens per second.:latency: lowest time to first token.:price/:cheap: lowest estimated cost for the request’s input and expected output, including selection markup. Opt-in providers that are otherwise used only when selected explicitly are included; when Auto routing is no more expensive, the request stays on Auto and is billed at the Auto price.:floor: alias for:price.:nitro: alias for:throughput.:tools: route to a tools-capable provider path when the model supports tools provider selection.
- Routing preference suffixes are billed like explicit provider-selection requests.
- They cannot be combined with
X-Provider, bodyprovider, or provider model suffixes. :toolscannot be combined with routing preference suffixes orcaching: true.- If the model does not support provider selection, the request returns an invalid request error for these suffixes.
Persistent Provider Preferences
These endpoints let a user save provider preferences in their session metadata.X-Provider, unless preferences were already saved on the associated user session.
Saved provider preferences apply to pay-as-you-go traffic. For subscription-included traffic, send X-Billing-Mode: paygo or billing_mode: "paygo" when you want saved provider preferences to apply.
Example GET response (placeholders):
PATCH payload:
preferredProviders: ordered list of allowed providers; the system tries each in order.excludedProviders: providers that should never be used.enableFallback: whentrue(default), permits retries among eligible providers. A configuredpreferredProviderslist remains a hard allowlist; fallback does not escape it to the platform default.modelOverrides: optional per-model overrides for the fields above.availableProviders: full set of provider IDs available to your account for the model.
Resolution Order
Whencaching: true is set without a routing preference, use the routing order in Cache-Capable Provider Routing. Otherwise, when multiple selections exist, the system resolves providers in this order:
- Per-request explicit provider selection (
X-Provider, bodyproviderstring/object, or provider suffix) - Per-request routing preference suffix (
:fast,:cheap, etc.) - Per-model
preferredProviders - Global
preferredProviders - Platform default when no binding provider allowlist or request constraint selects a route
Billing and Markup
- If you explicitly select or constrain providers with
X-Provideror bodyprovider, billing uses the resolved provider-specific price plus the applicable provider-selection markup (normally 5%) and the request is treated as pay-as-you-go. - If saved preferences select a provider on pay-as-you-go traffic, billing uses the provider-specific price plus the applicable provider-selection markup (normally 5%).
- If you do not select a provider, billing uses the model’s default price.
provider.preferred_max_latencyandprovider.preferred_min_throughputdon’t change billing by themselves.
Which Provider Served My Request?
The Usage page shows a label such as Via Cerebras under the model name, and a Provider row in request details, when public attribution is available. This can include explicit selections, supported BYOK routes, or a public provider used as a fallback. Some automatic requests have no provider label. Chat responses and the request billing endpoint do not expose the upstream provider. Do not rely on an undocumented response header or infer a provider from timing or price.Error Behavior
-
400witherror.code: "provider_max_price_unverifiable"means an explicitly selected gateway has no verifiable provider price for the model. Choose a provider with published pricing, or remove the cap to use normal gateway billing. -
If
enableFallbackisfalseand no preferred provider is available,/api/v1/chat/completionsreturns400witherror.code: "no_fallback_available". -
Invalid body
providerobjects return400witherror.type: "invalid_request_error". -
PATCH /api/user/provider-preferencesvalidates provider IDs and returns:422 INVALID_INPUTfor malformed payloads.400 INVALID_EXCLUSIONSif exclusions would leave a model with no usable provider.