Skip to main content
This guide covers how to store, rotate, and manage provider API keys for your AI Gateway. Managing keys in the gateway keeps them out of your application code and enables features like automatic failover and intelligent key selection.
When you configure provider keys in the gateway, anyone with your endpoint URL can make requests using your provider keys. See Securing Your Gateway to add authorization.

​
Key storage options

​
Inline keys (development only)

Store keys directly in your Traffic Policy: 
providers:
  - id: "openai"
    api_keys:
      - value: "sk-proj-abc123..."
Inline keys are visible in Traffic Policy configurations. Use only for local development and testing. See ngrok Secrets for the recommended approach.
Store keys in ngrok Vaults & Secrets—they’re encrypted at rest, never visible in your Traffic Policy, and can be rotated without redeploying: 
providers:
  - id: "openai"
    api_keys:
      - value: ${secrets.get('openai', 'key-one')}
      - value: ${secrets.get('openai', 'key-two')}

​
Creating secrets

​
Using the CLI

ngrok api secrets create \
  --name openai \
  --secret-data '{"key-one": "sk-proj-abc123...", "key-two": "sk-proj-def456..."}'

​
Using the dashboard

  1. Navigate to Vaults & Secrets in the ngrok Dashboard
  2. Create a vault if you don’t have one (for example, ai-keys)
  3. Create a secret in the vault (for example, openai)
  4. Add your provider API keys as key-value pairs

​
Multiple keys for failover

Configure multiple provider API keys for automatic failover when keys hit rate limits or fail: 
providers:
  - id: "openai"
    api_keys:
      - value: ${secrets.get('openai', 'primary')}
      - value: ${secrets.get('openai', 'backup')}
      - value: ${secrets.get('openai', 'emergency')}
Failover behavior:
  1. Gateway tries the first key
  2. If it fails (rate limit, quota exceeded, error), tries the next key
  3. Continues until a key succeeds or all keys are exhausted
Key ordering matters: Keys are tried in the order listed. Place your highest-capacity keys first.

​
Key rotation

To rotate provider API keys without downtime:
1

Add the new key

api_keys:
  - value: ${secrets.get('openai', 'old-key')}
  - value: ${secrets.get('openai', 'new-key')}  # Added
2

Deploy and monitor

Deploy the updated Traffic Policy. Monitor traffic to ensure the new key works correctly.
3

Remove the old key

Once confirmed, remove the old key:
api_keys:
  - value: ${secrets.get('openai', 'new-key')}
4

Revoke the old key

Revoke the old key with your provider (OpenAI, Anthropic, etc.) to complete the rotation.

​
Multi-provider example

on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: "openai"
          api_keys:
            - value: ${secrets.get('openai', 'team-a-key')}
            - value: ${secrets.get('openai', 'team-b-key')}
            - value: ${secrets.get('openai', 'shared-backup')}
        
        - id: "anthropic"
          api_keys:
            - value: ${secrets.get('anthropic', 'primary')}
            - value: ${secrets.get('anthropic', 'secondary')}
        
        - id: "google"
          api_keys:
            - value: ${secrets.get('google', 'key')}
This configuration provides:
  • 3 failover keys for OpenAI
  • 2 failover keys for Anthropic
  • 1 key for Google
  • Automatic key rotation on failures

​
Intelligent key selection

For advanced control over which API key is used, configure api_key_selection with CEL expressions. This enables intelligent key rotation based on runtime metrics like quota usage and error rates.

​
Basic configuration

on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: openai
          api_keys:
            - value: ${secrets.get('openai', 'key-one')}
            - value: ${secrets.get('openai', 'key-two')}
            - value: ${secrets.get('openai', 'key-three')}
      
      api_key_selection:
        strategy:
          - "ai.keys.filter(k, k.quota.remaining_requests > 100)"
          - "ai.keys"

​
How it works

Strategies execute in order until one returns at least one key:
  1. First strategy filters keys with >100 remaining requests
  2. If no keys match, falls back to all keys
  3. Selected keys are then tried in order for failover

​
Quota-based selection

Prioritize keys with remaining capacity: 
api_key_selection:
  strategy:
    # High capacity keys first
    - "ai.keys.filter(k, k.quota.remaining_requests > 500)"
    # Medium capacity keys
    - "ai.keys.filter(k, k.quota.remaining_requests > 100)"
    # Any key with quota
    - "ai.keys.filter(k, k.quota.remaining_requests > 0)"
    # Fall back to all keys
    - "ai.keys"

​
Error rate-based selection

Avoid keys experiencing issues: 
api_key_selection:
  strategy:
    # Keys with very low rate limit errors
    - "ai.keys.filter(k, k.error_rate.rate_limit < 0.05)"
    # Keys with acceptable error rates
    - "ai.keys.filter(k, k.error_rate.total < 0.2)"
    # Fall back to all keys
    - "ai.keys"

​
Load distribution

Randomize key selection to distribute load: 
api_key_selection:
  strategy:
    # Randomly select from keys with good quota
    - "ai.keys.filter(k, k.quota.remaining_requests > 100).randomize()"
    # Fall back to any key
    - "ai.keys.randomize()"

​
Available key variables

VariableDescription
k.quota.remaining_requestsRequests remaining before rate limit
k.quota.remaining_tokensTokens remaining before rate limit
k.error_rate.totalFraction of all errors (0.0 to 1.0)
k.error_rate.rate_limitFraction of rate limit (429) errors
k.error_rate.timeoutFraction of timeout errors
See CEL Functions Reference for the complete list.

​
Security best practices

  • Add authorization to your gateway when using server-side keys—see Securing Your Gateway
  • Never commit provider API keys to version control
  • Use ngrok secrets for all production keys
  • Rotate keys regularly to minimize exposure risk
  • Monitor key usage to detect anomalies
  • Use different keys for different environments (dev, staging, prod)
  • Set up alerts for rate limit errors to proactively add capacity

​
Passthrough mode

If you don’t configure any keys for a provider, the gateway forwards whatever key your SDK sends: 
providers:
  - id: "openai"
    # No api_keys - SDK key is forwarded
This is useful for:
  • Development environments where each developer uses their own key
  • Applications that already manage their own keys
  • Quick testing without configuration changes

​
Next steps