Skip to main content
When an upstream attempt fails, the AI Gateway either moves to another candidate or stops and returns the error. This page explains which failures it retries, in what order, and what bounds the retrying.

​
What triggers failover

The gateway moves to another candidate on these failures:
  • Authentication failures: HTTP 401 or 403 from the provider, which usually means the credential is invalid or exhausted.
  • Rate limits: HTTP 429.
  • Server errors: HTTP 500 and above.
  • Request timeouts: HTTP 408, or the attempt exceeding your per-request timeout.
  • Network failures: connection errors, DNS failures, and TLS errors.
Any other 4xx response stops the request immediately. A malformed request or an unsupported parameter is not going to succeed on a different provider, so the gateway returns the error rather than spending your quota on it.

​
Failover order

Where the gateway goes next depends on how the attempt failed. Model candidates come from the model field first, then the models array in order:
The gateway never retries the same model and key combination, except in the one case below.
Cross-provider failover needs candidates from more than one provider. List them in the models array, or set them up as fallback models.

​
Retrying the same provider

When a provider answers 429 or 503 with a Retry-After header, the gateway waits and retries the same backend instead of moving on. It does that only when the wait is 5 seconds or less, only when the wait still fits inside your total timeout, and at most twice per request. Otherwise it treats the response as a rate limit or a server error and moves to the next candidate.

​
What bounds the retrying

Two timeouts and an attempt cap bound every request: Set the timeouts for your account in Account Settings, or through the Configuration API. When the total timeout is reached, failover stops even if candidates remain, and the gateway returns ERR_NGROK_3810.

​
Failures that never reach failover

Some requests fail before the gateway makes any upstream attempt. These return immediately, and the response carries no Ngrok-AIG-Attempts header:
  • The request body could not be parsed.
  • The model is unknown, or the model address is malformed.
  • The requested provider or model is blocked by the access key configuration’s access scope.
  • No provider key is available for the requested provider.
  • The account has no credits and the request needs an ngrok-managed key.
None of these are billed. The gateway bills an attempt only once its response reaches your client, so a request that never reached a provider costs nothing. A stream that fails partway through is billed, because its first bytes were already delivered. See Error Codes for the code each of these returns.

​
Best practices

  1. Attach several provider keys per provider so an exhausted credential fails over instead of failing the request. See Multi-key failover.
  2. List candidates from more than one provider so a provider-wide outage has somewhere to go.
  3. Set timeouts to match your latency budget. A total timeout that is too tight cuts failover short; one that is too loose makes a slow failure look like a hang.
  4. Record Ngrok-AIG-Request-Id from every response so you can look a failure up afterwards. See Debug failed requests.

​
Next steps

Error Codes

Error codes, causes, and solutions

Debug failed requests

Diagnose a failure from the response