Vercel AI Gateway Private Inference: what to check first
Vercel AI Gateway Private Inference puts your own model endpoints behind one API. How setup, fees and fallbacks work, plus a security checklist.
Vercel AI Gateway Private Inference shipped on November 16, 2026. You can now register your own model endpoint (a vLLM server, a fine-tune, a prerelease model) and call it through the gateway with the same API keys, SDKs and logs you already use for catalog models like Claude or GPT. You address it as private/<slug>.
I have been on the self-hosting side of this problem for a while. When a client's NDA says customer data can't go to a third-party model API, you end up running your own model, and then you maintain a second set of keys, a second client and a second place to look at logs. This feature is aimed at that gap. Its fallback and billing design also affects data residency, and that is the part I want to look at closely.
What Vercel AI Gateway Private Inference does
According to the changelog post and the Private Inference docs, there are two building blocks:
- A private provider: a base URL, the API format the endpoint speaks, and an API key that the gateway sends to your endpoint on every request. The key is stored encrypted and only the last four characters are shown after saving.
- A private model: a slug, the upstream model ID your server expects, and a type (language, embedding or image).
The gateway translates between formats. Your app can send a Chat Completions, Responses or Messages request, and the gateway converts it to whatever your endpoint speaks. Switching between a catalog model and your own model only changes the model ID. Only the OpenAI Chat Completions format serves embedding and image models; the other formats are language only.
Setup works from the dashboard or the Vercel CLI (65.16.0 or later):
vercel ai-gateway private-inference providers create acme-vllm \
--base-url https://vllm.example.com/v1
vercel ai-gateway private-inference models create support-fine-tune \
--provider acme-vllm \
--api-model acme/support-llama-ft
Then you call it like any other model:
import { generateText } from 'ai';
const { text } = await generateText({
model: 'private/support-fine-tune',
prompt: 'Draft a reply to a customer who was charged twice.',
});
Limits: up to 25 private providers and 100 private models per team. Vercel also says private models work in coding agents connected to the gateway, such as Claude Code, Codex and Cursor.
This is different from Bring Your Own Key. BYOK is your credentials for a provider the gateway already supports. Private Inference adds an endpoint and a model the gateway wouldn't know about otherwise.
Your endpoint still has to be on the public internet
The base URL must be a publicly reachable HTTPS endpoint. It can't embed credentials, use a non-standard port, or resolve to a private, loopback or link-local address.
That blocks the obvious server-side request forgery trick of pointing the gateway at internal addresses. For you, it means the vLLM box you used to keep inside a private network now needs a public hostname on port 443. That changes your threat model.
What I would do before connecting anything:
- Put real authentication in front of the model server. The gateway sends an API key on every request, so check it at a reverse proxy, and make it a long random value used only for this provider.
- Add rate limits at that proxy. A public inference endpoint is an expensive GPU anyone can try to hit.
- Plan rotation. You can change a provider's API key on its page, so treat it like any other service credential.
- Send a test request right away. Adding a provider doesn't test the connection, so a typo in the base URL only shows up on the first call.
Also note that any team member can view private providers and models. Creating, editing and deleting needs the Member or Owner role. If your Vercel team includes contractors, check who those people are.
Zero Data Retention is a self-attestation
Each provider has a Zero Data Retention toggle. The docs say to turn it on only if your endpoint doesn't retain request data, and that any Member or Owner can attest it. Vercel takes your team's word for it. If your vLLM deployment writes prompts to disk or a log pipeline, the toggle doesn't change that. Audit your server's logging before you flip it.
Fallbacks, fees and data residency
Here is the part I would read twice.
You can give a private model a fallback to a catalog model. If your endpoint fails, the request goes to, say, openai/gpt-5.4-mini on Vercel's provider keys. Pricing pushes you toward doing that:
| Private model request | AI Gateway fee |
|---|---|
| Catalog fallback in the chain, Fallback to Vercel Keys on | No additional cost |
| No catalog fallback in the chain | $0.20 per 1,000 requests |
| Fallback to Vercel Keys turned off | $0.20 per 1,000 requests |
Vercel doesn't bill tokens on private models; you pay your own provider for the compute. At list price, a million fee-bearing requests cost $200. Failed requests aren't charged.
The trade-off is plain. If you self-host because a contract says customer data stays off third-party APIs, a catalog fallback breaks that promise exactly when your server is down. In my work that is the common case: the private model exists for compliance reasons. For those workloads I would leave the fallback out and pay the fee. For a fine-tune that only exists for quality or cost reasons, the free fallback is a good deal.
Whatever you choose, write it down per model. Your security review and your client should both know which models may fall back to a third-party provider.
Where it fits
For teams already on AI Gateway, this removes glue code. Private traffic shows up in the same logs and trace drains, with charts for requests, tokens and time to first token per private model. That makes A/B tests between a self-hosted model and a catalog model much simpler. My post on self-hosting Mistral Large 4 covered running the model; this covers how your apps reach it.
What it doesn't remove: you still run and secure the model server. And your prompts now pass through Vercel's gateway on the way to your own server, so the gateway becomes part of your data path. If your agents use private models, the same rules from securing AI agents apply: scoped credentials, logged actions, and limits on what the agent can reach.
My plan: connect a staging vLLM endpoint behind an authenticating proxy, run a set of real tasks through private/ and a catalog model side by side, and decide fallback policy model by model, in writing.
FAQ
What API formats does Private Inference support?
OpenAI Chat Completions, OpenAI Responses, Open Responses and Anthropic Messages. Vercel suggests Open Responses for self-hosted servers like vLLM and LM Studio when in doubt.
Can I connect a model on a private network?
No. The base URL must be publicly reachable over HTTPS and can't resolve to a private, loopback or link-local address.
How much does Vercel AI Gateway Private Inference cost?
No token charges from Vercel. There is a $0.20 per 1,000 requests fee unless the request includes a catalog model fallback with Fallback to Vercel Keys on. You need purchased AI Gateway Credits.
Sources: Vercel changelog: AI Gateway now supports Private Inference, Vercel docs: AI Gateway Private Inference
Cover image: Vercel, AI Gateway Private Inference announcement
Related articles
Qwen4-Max API pricing: 1M context, closed weights
Qwen4-Max API pricing is $1.50/$4.50 per million tokens with 1M context. What the cache discount, benchmarks and closed weights mean for agent builders.
Securing AI agents: lessons from Uber's Secure AI framework
Securing AI agents takes more than API keys. How Uber's Secure AI framework checks if an agent's action fits its purpose, and what small teams can copy.
Mistral Large 4 open weights: what it takes to self-host
Mistral Large 4 open weights are out: a 1T-parameter MoE model for coding and security. Benchmarks, license terms, API pricing and hardware to self-host it.