Skip to main content

Documentation Index

Fetch the complete documentation index at: https://ngrok.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

Groq provides high-speed inference for open-source models (Llama, Mixtral, and others) using their custom LPU hardware. Groq requires you to bring your own key—ngrok-managed keys are not available for Groq.

​
Setup

1

Create an AI Gateway endpoint

If you don’t have one yet, follow the quickstart to create your AI Gateway endpoint.
2

Get a Groq API key

Sign up at console.groq.com and create an API key. Keys start with gsk_.
3

Make a request

Pass your key directly—the gateway forwards it to Groq.
from openai import OpenAI

client = OpenAI(
    base_url="https://your-ai-gateway.ngrok.app/v1",
    api_key="gsk_..."  # Your Groq key, forwarded by gateway
)

response = client.chat.completions.create(
    model="meta-llama/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

​
Available models

See the model catalog for the full list of supported models.

​
Store key in the gateway

Instead of each client passing their own key, you can store it once in ngrok Secrets and have the gateway inject it automatically.
Storing your provider key in the gateway makes your endpoint publicly accessible. You must add authorization to prevent unauthorized use and unexpected charges. See Protecting BYOK Endpoints.
1

Store your key in ngrok secrets

ngrok api secrets create \
  --name groq \
  --secret-data '{"api-key": "gsk_..."}'
Or use the Vaults & Secrets dashboard.
2

Configure your traffic policy

on_http_request:
  - type: ai-gateway
    config:
      providers:
        - id: "groq"
          api_keys:
            - value: ${secrets.get('groq', 'api-key')}
3

Make a request

Clients no longer need a Groq key—pass any value for api_key.
from openai import OpenAI

client = OpenAI(
    base_url="https://your-ai-gateway.ngrok.app/v1",
    api_key="unused"  # Gateway injects your Groq key
)

response = client.chat.completions.create(
    model="meta-llama/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

​
Next steps