Create embeddings for search and retrieval through AI Gateway

Your embedding requests can now go through the same AI Gateway URL and access key as your chat requests. Managed models bill to your ngrok credits, so you can try OpenAI, Qwen, Tencent, and Gemini embeddings without a separate account for each provider. Every request shows up in Logs with its token usage and cost, and switching models on /v1/embeddings is a change to one string.

An embedding turns text into a list of numbers. Texts with similar meanings get similar vectors, so your application can compare them to find the closest match. That’s the building block for semantic search, retrieval for RAG, deduplication, and classification. The example below uses it to match a production alert to the runbook most likely to fix it.

Models and pricing

Embedding requests bill input tokens only, because a vector has no output tokens. These models work on the OpenAI-format /v1/embeddings endpoint:

Priced models bill to your ngrok credits. For models marked Your own key, attach your Mistral AI or QwenCloud key, and the provider bills those requests directly.

Gemini embedding models use Google’s native request format instead. Call /v1beta/models/gemini-embedding-2:embedContent, or :batchEmbedContents for several inputs, with Google’s Gen AI SDK. gemini-embedding-2 bills to your credits at $0.20 per million input tokens. gemini-embedding-001 requires your own Google key. The Google Gen AI SDK guide covers the client setup.

A few limits apply to embedding requests:

• ngrok/auto isn’t available, because automatic routing picks among chat models.
• A fallback list must name one model. Vectors from different models don’t compare, so the gateway refuses a list that mixes models with a 400 instead of corrupting your index.
• Requests accept text input only. Streaming isn’t supported.

Get started

Create an ngrok account or sign in. Create an access key that allows OpenAI, and confirm your account has credits.

Follow the setup guide and select OpenAI and TypeScript.

Install the OpenAI SDK and TypeScript runner:

npm install openai && npm install --save-dev tsx

Save your ngrok access key in .env.local. Keep this file out of version control:

AI_GATEWAY_API_KEY=your_access_key_here

This example matches a production alert to the runbook that’s most likely to fix it. It embeds the alert and three runbook titles in one request, then compares the alert’s vector against each runbook’s vector with cosine similarity. The closer the score is to 1, the closer the meaning.

Use an ES module project ("type": "module" in package.json) for top-level await. Save this example as index.ts:

1import OpenAI from "openai";2 3const client = new OpenAI({⋯4	baseURL: "https://gateway.ngrok.ai/v1",5	apiKey: process.env.AI_GATEWAY_API_KEY,6});7 8const alert = "checkout-api: 5xx rate at 38% since the latest deploy";9const runbooks = [⋯10	"Roll back a bad deploy of the checkout service",11	"Renew an expired TLS certificate",12	"Add database capacity when connections run out",13];14 15const response = await client.embeddings.create({⋯16	model: "openai/text-embedding-3-small",17	input: [alert, ...runbooks],18	encoding_format: "float",19});20 21const vectors = response.data.map((item) => item.embedding);22const alertVector = vectors[0];23 24function cosineSimilarity(a: number[], b: number[]) {⋯25	let dot = 0;26	let normA = 0;27	let normB = 0;28	a.forEach((value, i) => {⋯29		dot += value * b[i];30		normA += value * value;31		normB += b[i] * b[i];32	});33	return dot / (Math.sqrt(normA) * Math.sqrt(normB));34}35 36const ranked = runbooks37	.map((title, i) => ({ title, score: cosineSimilarity(alertVector, vectors[i + 1]) }))38	.sort((a, b) => b.score - a.score);39 40console.log(`Best match: ${ranked[0].title} (${ranked[0].score.toFixed(2)})`);41console.log(`Input tokens: ${response.usage.prompt_tokens}`);

Run the example from your project directory:

node --env-file=.env.local --import tsx index.ts

response.data holds one embedding per input, in the order you sent them. response.usage.prompt_tokens is the token count the request bills on. The output looks something like this. The score and token count shown here are illustrative and vary by model:

1Best match: Roll back a bad deploy of the checkout service (0.58)2Input tokens: 42

In a real application, embed your documents once, store the vectors, and embed only the incoming query at request time.

To try another model, change model to one of the IDs above. Keep encoding_format: "float". The OpenAI SDK asks for base64 by default, and the QwenCloud and TokenHub models return float vectors only. text-embedding-3-small, text-embedding-3-large, and the QwenCloud models also accept dimensions for shorter vectors.

Open Logs, find the request you just sent, and inspect its model, provider, status, token usage, and cost.