Skip to main content

Documentation Index

Fetch the complete documentation index at: https://ngrok.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

Azure OpenAI Service provides access to OpenAI models through Microsoft’s Azure cloud. This guide shows you how to connect Azure OpenAI to the ngrok AI Gateway.

​
Prerequisites

​
Overview

Azure OpenAI uses a different URL structure than standard OpenAI. You configure the full deployment URL as the base_url and the gateway routes requests to it.

​
Getting started

1

Get Azure OpenAI details

From the Azure Portal, gather:
  1. Endpoint URL: https://your-resource.openai.azure.com
  2. API Key: From “Keys and Endpoint” section
  3. Deployment Name: The name you gave your model deployment
2

Store your API key

Add your Azure OpenAI API key to ngrok secrets:
ngrok api secrets create \
  --name azure-openai \
  --secret-data '{"api-key": "your-azure-api-key"}'
You can also create secrets in the ngrok Dashboard.
3

Configure the AI Gateway

Create a Traffic Policy with Azure OpenAI as a provider:
policy.yaml
on_http_request:
  - type: ai-gateway
    config:
      headers:
        api-version: "2024-02-15-preview"
      providers:
        - id: "azure-openai"
          base_url: "https://your-resource.openai.azure.com/openai/deployments/your-deployment"
          api_keys:
            - value: ${secrets.get('azure-openai', 'api-key')}
          models:
            - id: "gpt-4o"
Azure OpenAI requires the api-version header. The headers configuration above ensures this is added to all requests.
4

Use with OpenAI SDK

Point any OpenAI-compatible SDK at your AI Gateway:
from openai import OpenAI

client = OpenAI(
    base_url="https://your-ai-gateway.ngrok.app/v1",
    api_key="unused"  # Gateway handles auth
)

response = client.chat.completions.create(
    model="azure-openai:gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

​
Advanced configuration

​
Multiple deployments

Configure multiple Azure OpenAI deployments: 
on_http_request:
  - type: ai-gateway
    config:
      headers:
        api-version: "2024-02-15-preview"
      providers:
        - id: "azure-gpt4"
          base_url: "https://your-resource.openai.azure.com/openai/deployments/gpt-4o-deployment"
          api_keys:
            - value: ${secrets.get('azure-openai', 'api-key')}
          models:
            - id: "gpt-4o"
        
        - id: "azure-gpt35"
          base_url: "https://your-resource.openai.azure.com/openai/deployments/gpt-35-deployment"
          api_keys:
            - value: ${secrets.get('azure-openai', 'api-key')}
          models:
            - id: "gpt-3.5-turbo"

​
Multiple regions

Configure multiple Azure regions for failover: 
on_http_request:
  - type: ai-gateway
    config:
      headers:
        api-version: "2024-02-15-preview"
      providers:
        - id: "azure-eastus"
          base_url: "https://myapp-eastus.openai.azure.com/openai/deployments/gpt-4o"
          api_keys:
            - value: ${secrets.get('azure-openai-eastus', 'api-key')}
          models:
            - id: "gpt-4o"
          metadata:
            region: "eastus"
        
        - id: "azure-westus"
          base_url: "https://myapp-westus.openai.azure.com/openai/deployments/gpt-4o"
          api_keys:
            - value: ${secrets.get('azure-openai-westus', 'api-key')}
          models:
            - id: "gpt-4o"
          metadata:
            region: "westus"
Without a model selection strategy, requesting model: "gpt-4o" returns both regions as candidates (in config order), enabling failover. Requesting model: "azure-eastus:gpt-4o" pins to that region only. For explicit control over failover order, clients can use models: ["azure-eastus:gpt-4o", "azure-westus:gpt-4o"].

​
Failover to OpenAI

Use Azure as primary with OpenAI fallback: 
on_http_request:
  - type: ai-gateway
    config:
      headers:
        api-version: "2024-02-15-preview"
      providers:
        - id: "azure-openai"
          base_url: "https://your-resource.openai.azure.com/openai/deployments/gpt-4o"
          api_keys:
            - value: ${secrets.get('azure-openai', 'api-key')}
          models:
            - id: "gpt-4o"
        
        - id: "openai"
          api_keys:
            - value: ${secrets.get('openai', 'api-key')}
      
      model_selection:
        strategy:
          - "ai.models.filter(m, m.provider_id == 'azure-openai')"
          - "ai.models.filter(m, m.provider_id == 'openai')"
The first strategy that returns models wins. If Azure has matching models, only those are tried. OpenAI is only used if no Azure models match. For cross-provider failover when requests fail, have clients specify multiple models: models: ["azure-openai:gpt-4o", "openai:gpt-4o"].

​
Embeddings

Configure Azure OpenAI embeddings: 
on_http_request:
  - type: ai-gateway
    config:
      headers:
        api-version: "2024-02-15-preview"
      providers:
        - id: "azure-embeddings"
          base_url: "https://your-resource.openai.azure.com/openai/deployments/text-embedding-ada-002"
          api_keys:
            - value: ${secrets.get('azure-openai', 'api-key')}
          models:
            - id: "text-embedding-ada-002"

​
Troubleshooting

​
401 unauthorized

Symptom: Requests fail with authentication errors. Solutions:
  1. Verify the API key is correct in secrets
  2. Check the key hasn’t been regenerated in Azure Portal
  3. Ensure the secret name matches your config

​
404 deployment not found

Symptom: Requests fail with deployment not found. Solutions:
  1. Verify the deployment name in your base_url
  2. Check the deployment exists in Azure Portal
  3. Ensure the deployment is in the correct region

​
API version errors

Symptom: Requests fail with API version errors. Solutions:
  1. Update the api-version header to a supported version
  2. Check Azure OpenAI API versions for current versions

​
Rate limiting

Symptom: 429 errors from Azure. Solutions:
  1. Configure multiple deployments for failover
  2. Request quota increase in Azure Portal
  3. Add multiple API keys per deployment for automatic failover

​
Next steps