Add authorization to an API service but share docs/specs openly

1. Create two internal agent endpoints

On a server for your API service, create an internal agent endpoint.

ngrok http 8080 --url https://api-service.internal

On a server for your API documentation, create an internal agent endpoint.

ngrok http 8081 --url https://api-docs.internal

2. Reserve a domain

Navigate to the Domains section of the ngrok dashboard and click New + to reserve a domain like https://your-api.ngrok.app, which we’ll refer to $YOUR_NGROK_DOMAIN from here on out.

3. Create a cloud endpoint

Navigate to the Endpoints section of the ngrok dashboard, then click New + and Cloud Endpoint to create a new cloud endpoint on $YOUR_NGROK_DOMAIN.

4. Apply Traffic Policy to your cloud endpoint

You might need to change:

• /docs : The path where you'd like to make API documentation/specs available to unauthenticated requests from developers' web browsers.

on_http_request:
  - expressions:
      # Check whether the client's IP is on a known blocklist in IP Intelligence; if so, deny their request
      - "conn.client_ip.is_on_blocklist == true" 
    actions: 
      - type: custom-response 
        config: 
          content: "Unauthorized request"
          status_code: 403 
  - expressions:
      # Populate a `robots.txt` file to attempt to turn AI and search bots away from crawling your endpoint
      - "req.url.contains('/robots.txt')"
    actions:
      - type: custom-response 
         config: 
           status_code: 200 
           content: "User-agent: *\r\nDisallow: /" 
           headers: 
             content-type: text/plain
  - expressions:
      - "req.url.path.contains('/docs')"
    actions:
      - type: forward-internal
        config:
          url: https://api-docs.internal
  - expressions:
      # Allow all requests to the `/docs` path through without authentication, but require Basic Auth for all other paths
      - "!req.url.path.contains('/docs')"
    actions:
      - type: basic-auth
       config:
         # Add up to 10 pairs of username and password pairs
         credentials:
           - user1:password1
           - user2:password2
           - user3:password3
  - expressions:
      # Check whether the request contains valid Basic Auth credentials
      - "(actions.ngrok.basic_auth.credentials.authorized) && !req.url.path.contains('/docs')"
    actions:
      - type: forward-internal
        config:
          url: https://api-service.internal
  - expressions:
      # Deny all requests to without valid Basic Auth credentials
      - "(!actions.ngrok.basic_auth.credentials.authorized) && !req.url.path.contains('/docs')"
    actions:
      - type: custom-response
        config:
          content: Unauthorized
          status_code: 400

With these endpoints and Traffic Policy rules, you block AI/search bots from accessing your endpoint at all and allow developers to read your API documentation or specs without authentication. For all other paths, like /api/example , clients need to supply a valid username and password with their request.