Skip to main content
BlogNewsletterRSS
Back to changelog

Give your fallback models one name with virtual models

Shipped by Alice Wasko, Brenden Ehlers, Jarod Clark, Ben Chan, and Nijiko Yonskai

AI Gateway already fails over when your request sends a models list. That only works when you control the request body. A proxy that forwards bodies untouched, an SDK wrapper, or a tool with one model field can send a single model name and nothing else.

Virtual models move the list onto the gateway. Save up to 10 models in order under one name, such as rubber-duck, and send that name anywhere you’d put a model ID. The gateway tries each model in turn until one answers. Reorder the list once, and every client uses the new order on its next request. Nobody redeploys anything.

How it works

When a request’s model matches a virtual model’s name, the gateway swaps in its list. From there, the request works like one that sent models:

  • Each model still goes through your access key’s checks. A virtual model never lets a key call a model it can’t call directly. The gateway skips that model and tries the next one.
  • Failover rules don’t change. If a provider rejects the request itself, for example with a 400, the gateway stops. Your duck won’t carry a broken prompt to every provider you have.
  • Each request bills as the model that served it, and the Ngrok-AIG-Model response header names that model.
  • /v1/models lists each virtual model your key can use, so SDKs that check model names accept rubber-duck.

Limits

  • Names use up to 64 letters, numbers, ., _, and -. A name can’t match a model your account can already call.
  • A virtual model holds 1 to 10 models, with no repeats and no other virtual models. ngrok/auto works only as the sole entry.
  • An account can have up to 100 virtual models.
  • Access key configurations allow models, not virtual model names. To let a key use rubber-duck, allow its models.
  • If a model in the list stops resolving, such as a deleted custom model, the dashboard and API show a warning, and requests skip that model.

The virtual models guide covers the rest, including every warning and how a virtual model combines with a models list.

Get started

Create an ngrok account or sign in. Create an access key that allows Anthropic, OpenAI, and Google, and confirm your account has credits. The models in this example bill to your ngrok credits.

Open Virtual Models and click Create virtual model. Enter rubber-duck as the Name, click Add model for each model, and drag the rows into order. Click Create virtual model to save it.

To use the API instead, create an admin API key. It’s a different credential from an access key. Then send:

curl -X POST https://api.ngrok.ai/virtual-models \  -H "Authorization: Bearer <ADMIN_API_KEY>" \  -H "Content-Type: application/json" \  -d '{    "name": "rubber-duck",    "models": ["anthropic/claude-sonnet-5-5", "openai/gpt-6.1-sol", "google/gemini-3.8-flash"],    "description": "Patient debugging partner. Never judges."  }'

To change it later, send a PATCH to https://api.ngrok.ai/virtual-models/<ID>. A new models value replaces the whole list.

Next, follow the setup guide and select OpenAI and TypeScript. Install the OpenAI SDK and TypeScript runner:

npm install openai && npm install --save-dev tsx

Save your ngrok access key in .env.local. Keep this file out of version control:

AI_GATEWAY_API_KEY=your_access_key_here

Use an ES module project ("type": "module" in package.json) for top-level await. Save this example as index.ts. The only change from a normal request is the model value. withResponse() also returns the headers, so the script prints which model served:

import OpenAI from "openai"; const client = new OpenAI({	baseURL: "https://gateway.ngrok.ai/v1",	apiKey: process.env.AI_GATEWAY_API_KEY,}); const { data, response } = await client.chat.completions	.create({		model: "rubber-duck",		messages: [			{				role: "user",				content: "My tests pass locally and fail in CI. Ask me one question to narrow it down.",			},		],	})	.withResponse(); console.log(data.choices[0]?.message.content);console.log(`Served by: ${response.headers.get("ngrok-aig-model")}`);

Run the example from your project directory:

node --env-file=.env.local --import tsx index.ts

The output looks something like this. The question is illustrative and varies between runs:

Does CI run with the same environment variables as your machine?Served by: claude-sonnet-5-5

Now drag GPT-6.1 Sol to the top of the list and run the script again. Same code, different duck.

Open Logs, find the request, and inspect its status, model, provider, token usage, and cost. When the first model fails, Attempts shows each model the gateway tried.