Configuration Schema

Maximum number of tokens allowed in the prompt and context. Requests exceeding this limit will be rejected.

No limit is applied if not specified. Maximum allowed value is 500,000.

max_input_tokens: 4096

Maximum number of tokens allowed in the completion response.

No limit is applied if not specified. Maximum allowed value is 500,000.

max_output_tokens: 2048

Additional HTTP headers to include in requests to AI providers.

headers:
  X-Custom-Header: "value"
  X-Request-ID: "${req.id}"

Additional query parameters to append to provider requests.

query_params:
  api_version: "2023-10-01"

Additional JSON fields to merge into the request body.

body:
  temperature: 0.7
  top_p: 0.9

Behavior when all failover attempts are exhausted.

on_error: "continue"

Maximum total time for all failover attempts across all models and keys. Must be specified as a duration string (for example, “2m”, ”90s”).

total_timeout: "2m"

Timeout for a single request to a provider. Must be specified as a duration string (for example, ”45s”, “1m”).

per_request_timeout: "45s"

List of AI provider configurations. When empty, all built-in providers are allowed in passthrough mode.

See Provider Configuration below for detailed field definitions.

providers:
  - id: "openai"
    api_keys:
      - value: ${secrets.get('openai', 'key-one')}

When true, only providers explicitly listed in providers are allowed. Requests to other providers are rejected with an error.

only_allow_configured_providers: true

When true, only models explicitly listed in provider configurations are allowed. Requests for other models are rejected.

only_allow_configured_models: true

Strategy for selecting model candidates using CEL expressions. The first strategy that returns models wins—subsequent strategies are only used if previous ones return no models.

See Model Selection Strategies for details and CEL Functions Reference for available functions.

model_selection:
  strategy:
    - "ai.models.filter(m, m.provider_id == 'openai')"
    - "ai.models"

Strategy for selecting API keys using CEL expressions. Enables intelligent key selection based on metrics like quota usage and error rates.

When not specified, keys are tried in the order listed. See CEL Functions Reference for available functions.

api_key_selection:
  strategy:
    - "ai.keys.filter(k, k.quota.remaining_requests > 100)"
    - "ai.keys.filter(k, k.error_rate.rate_limit < 0.1)"
    - "ai.keys"

Provider identifier. Use built-in names (openai, anthropic, google, deepseek) or custom names for self-hosted providers.

- id: "openai"

Alternative identifiers for this provider. Allows clients to reference the same provider by different names.

- id: "custom-openai"
  id_aliases: ["openai", "gpt"]

Custom endpoint URL for self-hosted or alternative provider endpoints. Required for custom providers.

- id: "ollama"
  base_url: "https://ollama.internal.company.com"

Human-readable name for the provider.

Description of the provider.

Provider’s website URL.

Temporarily disable this provider without removing its configuration.

- id: "openai"
  disabled: true

Custom metadata for tracking and organization. Not sent to providers. Available in selection strategies via m.getMetadata().

- id: "openai"
  metadata:
    team: "ml-platform"
    environment: "production"

List of API keys for this provider. Keys are tried in order for automatic failover.

api_keys:
  - value: ${secrets.get('openai', 'primary')}
  - value: ${secrets.get('openai', 'backup')}

List of model configurations for this provider. See Model Configuration below.

The API key value. Use secrets.get() for secure storage.

api_keys:
  - value: ${secrets.get('openai', 'key-one')}

Model identifier as recognized by the provider.

models:
  - id: "gpt-4o"

Alternative identifiers for this model.

models:
  - id: "gpt-4o-2024-11-20"
    id_aliases: ["gpt-4o", "gpt-4-latest"]

ID of the model author (for third-party models).

Human-readable name for the model.

Description of the model.

Temporarily disable this model.

models:
  - id: "gpt-3.5-turbo"
    disabled: true

Custom metadata for the model. Available in selection strategies.

models:
  - id: "gpt-4o"
    metadata:
      tier: "premium"
      approved: true

Input types supported by the model (for example, “text”, “image”, “audio”).

Output types supported by the model.

Maximum context window size in tokens.

Maximum output tokens the model can generate.

Features supported by the model (for example, “tool-calling”, “coding”).