Introducing the ngrok AI gateway (early access)

Today we’re opening early access to the ngrok AI gateway—a single endpoint that routes your AI requests to OpenAI, Anthropic, Google, and any other provider you want, with automatic failover when things go wrong.

Bookmark this sectionThe problem

You’ve probably been here: your app hammers OpenAI’s API, hits a rate limit, and your users stare at spinning loaders while you scramble to add Anthropic as a backup. Or maybe you’re burning through API credits on GPT-4o when a cheaper model would work fine. Or you’re trying to route to your self-hosted Ollama instance during development but production runs against cloud providers.

All of these problems share the same root cause: your app talks directly to AI providers, and that means you’re stuck writing routing, failover, and retry logic yourself.

Bookmark this sectionA better approach

The AI gateway sits between your app and AI providers. Point your OpenAI SDK at your AI gateway, and we handle the rest:

1import OpenAI from "openai";2 3const client = new OpenAI({⋯4	baseURL: "https://your-gateway.ngrok.app/v1",5	apiKey: "your-api-key",6});7 8const response = await client.chat.completions.create({⋯9	model: "gpt-4o",10	messages: [{ role: "user", content: "Hello!" }],11});

That’s it. Your code stays the same. Behind the scenes, the gateway:

1. Receives your request
2. Selects which model and provider to use (based on your configuration)
3. Forwards the request with the appropriate provider API key
4. Retries with the next option if it fails
5. Returns the response

Bookmark this sectionWhat you can do

Use any SDK. We’re OpenAI-compatible, so if your SDK can set a baseURL, it works with us—that includes the official OpenAI SDK, Vercel AI SDK, LangChain, and pretty much everything else.

Failover across providers. Configure OpenAI as primary and Anthropic as backup. If OpenAI fails, we try Anthropic. You sleep better.

Rotate API keys. Use multiple API keys for the same provider. When one hits rate limits, we switch to another. Your app keeps running.

Define selection strategies. Control exactly how models get selected using CEL expressions. Prefer cheaper models? Lowest latency? Only models with tool-calling support? Write a one-liner:

1model_selection:⋯2  strategy:⋯3    # First, prioritise low latency.4    - "ai.models.filter(m, m.metrics.global.latency.upstream_ms_avg < 1000)"5    # If that gets no results, prioritise low cost.6    - "ai.models.sortBy('price')"

Route to self-hosted models. Send requests to Ollama, vLLM, LM Studio, or any OpenAI-compatible endpoint. Mix cloud and local inference in the same configuration.

Modify content in transit. Redact PII before requests hit AI providers. Sanitize responses on the way back. Inject system prompts. All in your traffic policy.

Bookmark this sectionWhat’s in early access

We’re shipping with:

• OpenAI-compatible API for routing to multiple providers
• Built-in support for OpenAI, Anthropic, Google, DeepSeek, OpenRouter, and more
• Automatic failover across models, providers, and API keys
• CEL-based model selection for custom routing logic
• Custom provider support for self-hosted models
• Token counting and usage metrics
• Traffic Inspector for debugging requests
• Log exporting to external systems

We’re actively working on:

• Dedicated AI gateway dashboard
• Provider and retry tracking
• Token usage visualization
• Cost analytics

Bookmark this sectionGetting started

1. Sign in at app.ngrok.ai with your existing ngrok credentials or a new account
2. Add credits on the Credits page
3. Create an access key on the Access Keys page
4. Point your SDK at https://gateway.ngrok.ai/v1
5. Send your first request

Check out the quickstart guide for the full walkthrough.

Bookmark this sectionWhat we’d love to hear

This is early access, which means we’re building this with you. We want to know:

• Which providers and models are you using?
• What selection strategies would be most useful?
• What observability features would actually help?
• What’s broken, confusing, or missing?

The feature set will evolve based on what you tell us. That’s not marketing speak—we genuinely don’t know exactly what this needs to become until we see how people use it.

Ready to tell us what you think? Reach out to the ngrok.ai team.

ngrok.ai is now an all-new experience. Sign up at app.ngrok.ai and read the docs for everything else.