Skip to main content
The AI Gateway can route requests to any OpenAI or Anthropic Claude API endpoint beyond the built-in providers, including self-hosted models, private deployments, and alternative AI services.

​
Requirements

Custom providers must:
  1. Expose an supported API - Same request/response format as OpenAI or Anthropic Claude
  2. Be reachable from ngrok - Either via HTTPS URL or ngrok internal endpoint
  3. Support the endpoints you need - /v1/chat/completions, /v1/embeddings, /v1/messages etc.

​
URL requirements

The base_url configuration has specific requirements based on the URL type:
URL TypeSchemeExample
External HTTPShttps:// onlyhttps://api.custom.com/v1
ngrok internal endpointhttp:// or https://https://my-service.internal
HTTP URLs are only allowed for ngrok .internal endpoints. All external URLs must use HTTPS.

​
External HTTPS URLs

For publicly accessible services with valid TLS certificates: 
providers:
  - id: "my-provider"
    base_url: "https://api.my-service.com/v1"
    api_keys:
      - value: ${secrets.get('my-provider', 'api-key')}

​
ngrok Internal Endpoints

For services running behind ngrok (local services, private networks), use internal endpoints: 
providers:
  - id: "my-local-service"
    base_url: "https://my-service.internal"
    # No API key needed if your service doesn't require one
Internal endpoints let you:
  • Route to local services without public exposure
  • Connect to other ngrok endpoints in your account
  • Use HTTP for services without TLS

​
Basic configuration

on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: "my-custom-provider"
          base_url: "https://my-ai-service.example.com/v1"
          supported_api_surfaces:
            - format: openai
            - format: anthropic
          api_keys:
            - value: ${secrets.get('custom', 'api-key')}
          models:
            - id: "my-model"

​
Configuration fields

FieldRequiredDescription
idYesUnique identifier for the provider
base_urlYesBase URL for the provider’s API
supported_api_surfacesNoAPI formats this provider supports, default is openai
api_keysNoAPI keys for authentication
modelsNoList of available models
id_aliasesNoAlternative names for the provider
metadataNoCustom metadata for selection strategies

​
Defining models

Specify which models your custom provider offers: 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    models:
      - id: "llama3-70b"
      - id: "mistral-7b"
      - id: "codellama-34b"

​
Model metadata

Add metadata to models for use in selection strategies: 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    models:
      - id: "llama3-70b"
        metadata:
          gpu: "A100"
          quantization: "none"
          max_context: 8192

​
Parameter filtering

By default, no parameter filtering occurs for custom providers or models. Every parameter your client sends is forwarded as-is. If your provider or model rejects parameters that clients commonly include, you can opt into filtering using either or both of the mechanisms below.

​
Provider surface allowlist

Use supported_params on a supported_api_surfaces entry to declare the exact set of parameters the provider accepts for that surface. Any top-level request body parameter not on this list will be removed before the request is forwarded. 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    supported_api_surfaces:
      - format: openai
        surface: chat-completions
        supported_params:
          - name: model
          - name: messages
          - name: temperature
          - name: max_tokens
          - name: stream
With this configuration, a request that includes logit_bias or parallel_tool_calls will have those fields removed before being forwarded to my-provider.
supported_params only takes effect when the gateway has matched a known surface (chat-completions, responses, or messages). The list must be non-empty to activate filtering.

​
Model-specific denylist

Use unsupported_params on a model to declare parameters that model doesn’t accept. These are removed regardless of what the provider surface allows, and regardless of which surface was matched. 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    models:
      - id: "my-model"
        unsupported_params:
          - name: parallel_tool_calls
          - name: response_format

​
Combining both

The two mechanisms are independent and can be used together. Provider-level filtering (allowlist) runs first, then model-level filtering (denylist) runs second. A common pattern is to set supported_params at the provider level for all models, then use unsupported_params on individual models that have additional restrictions: 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    supported_api_surfaces:
      - format: openai
        surface: chat-completions
        supported_params:
          - name: model
          - name: messages
          - name: temperature
          - name: max_tokens
          - name: stream
    models:
      - id: "my-base-model"
        # No extra filtering needed—provider surface allowlist is sufficient
      - id: "my-limited-model"
        # This model additionally can't handle temperature even though the surface allows it
        unsupported_params:
          - name: temperature

​
Authentication

​
With provider API keys

If your service requires authentication: 
providers:
  - id: "my-provider"
    base_url: "https://my-service.example.com"
    api_keys:
      - value: ${secrets.get('my-provider', 'api-key')}

​
Without provider API keys

Some self-hosted services don’t require authentication: 
providers:
  - id: "my-provider"
    base_url: "https://my-service.internal"
    # No api_keys needed

​
Custom headers

For services requiring non-standard authentication: 
on_http_request:
  - type: ai-gateway
    config:
      headers:
        X-Custom-Auth: ${secrets.get('my-provider', 'token')}
      providers:
        - id: "my-provider"
          base_url: "https://my-service.example.com"

​
Timeouts

Self-hosted models can be slower than cloud providers. Adjust timeouts as needed: 
on_http_request:
  - type: ai-gateway
    config:
      per_request_timeout: "120s"
      total_timeout: "5m"
      providers:
        - id: "my-provider"
          base_url: "https://my-service.internal"

​
Restricting access

Allow only your custom provider and block cloud providers: 
on_http_request:
  - type: ai-gateway
    config:
      only_allow_configured_providers: true
      only_allow_configured_models: true
      providers:
        - id: "my-provider"
          base_url: "https://my-service.internal"
          models:
            - id: "llama3"
            - id: "mistral"

​
Failover patterns

​
Custom provider with cloud fallback

Use your self-hosted model as primary with cloud backup: 
on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: "my-provider"
          base_url: "https://my-service.internal"
          models:
            - id: "llama3"
        
        - id: "openai"
          api_keys:
            - value: ${secrets.get('openai', 'api-key')}
      
      model_selection:
        strategy:
          - "ai.models.filter(m, m.provider_id == 'my-provider')"
          - "ai.models.filter(m, m.provider_id == 'openai')"
The first strategy that returns models wins. If your custom provider has matching models, only those are tried. OpenAI is only used if no custom provider models match. For cross-provider failover when requests fail, have clients specify multiple models: models: ["my-provider:llama3", "openai:gpt-4o"].

​
Multiple custom providers

Load balance across multiple self-hosted instances: 
on_http_request:
  - type: ai-gateway
    config:
      only_allow_configured_providers: true
      providers:
        - id: "inference-1"
          base_url: "https://inference-1.internal"
          models:
            - id: "llama3"
        
        - id: "inference-2"
          base_url: "https://inference-2.internal"
          models:
            - id: "llama3"
      
      model_selection:
        strategy:
          # Randomize across configured providers only
          - "ai.models.random()"

​
Troubleshooting

​
Connection refused

  • Verify the base_url is correct and reachable
  • For internal endpoints, ensure the ngrok tunnel is running
  • Check firewall rules allow traffic

​
HTTPS required error

External URLs must use HTTPS. For local services, use ngrok internal endpoints: 
# Expose local service with internal endpoint
ngrok http 8000 --url https://my-service.internal

​
Authentication errors

  • Verify API key is correct in secrets
  • Check if the service requires specific headers
  • Some services use Authorization: Bearer vs Api-Key

​
Model not found

  • Verify the model ID matches exactly what the service expects
  • Check if the model is loaded/downloaded on the service
  • Some services require specific model name formats

​
Integration guides

Step-by-step setup instructions for specific platforms:

Ollama

Run open-source models locally with Ollama

LM Studio

Desktop app for local model inference

vLLM

High-performance inference server

Azure OpenAI

Microsoft’s OpenAI service