Skip to main content
The AI Gateway provides observability at multiple levels:

Metrics for Model Selection

Real-time latency and error rate metrics available in CEL expressions for intelligent routing decisions.

Traffic Inspector

View full request and response bodies for debugging in the ngrok dashboard.

Log Exporting

Export HTTP logs to external systems for logging and analysis.
The AI Gateway is in Early Access. We’re actively working on additional observability features including dedicated dashboards, provider/retry tracking, token usage visualization, and cost analytics.Have feedback? We’d love to hear what observability features would be most valuable to you. Email [email protected] with your ideas.

​
Current capabilities

FeatureStatusDescription
CEL Metrics✅ AvailableLatency, error rates, token counts available in model_selection strategies
Traffic Inspector✅ AvailableStandard HTTP request/response inspection
Log Exporting✅ AvailableStandard ngrok HTTP traffic logs
AI-specific Events🚧 RoadmapProvider selection, retry tracking, token usage events
Dedicated Dashboard🚧 RoadmapAI Gateway-specific analytics and visualizations

​
Quick example: Metrics-based routing

Use real-time metrics to route to the fastest, most reliable models: 
on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: openai
          api_keys:
            - value: ${secrets.get('openai', 'key')}
        - id: anthropic
          api_keys:
            - value: ${secrets.get('anthropic', 'key')}
      model_selection:
        strategy:
          # Prefer low-latency models with good reliability
          - "ai.models.filter(m, m.metrics.global.latency.upstream_ms_avg < 1000 && m.metrics.global.error_rate.total < 0.05)"
          # Fallback to any available model
          - "ai.models"
Then connect using the OpenAI SDK or an SDK of your choice.

​
Understanding metric scope

Metrics are only available within model_selection strategy expressions. They are not available in general CEL expressions elsewhere in your Traffic Policy.This is because metrics are populated at runtime during AI Gateway request processing, specifically when evaluating model selection strategies.
For full details on available metrics and their scopes, see Metrics Reference.

​
Coming soon

Enhanced observability features are actively being developed:
  • AI Gateway Dashboard - Dedicated views for provider performance, token usage, and costs
  • Provider Tracking - See which providers handle each request
  • Retry Visibility - Track failover decisions and retry attempts
  • Token Analytics - Visualize token usage trends across models
  • Cost Estimation - Automatic cost calculation based on provider pricing
Want to influence the roadmap? Email [email protected] with what observability features you need.