Introducing IP Intelligence to control unwanted traffic

Let’s face it: Any time you put something on the internet, you’re opening it up to be discovered and crawled by anyone at any time. From bots and scrapers to highly organized global attackers, the reality is that unless you have unlimited bandwidth and compute power, this unwanted traffic costs you.

At ngrok, we’re in the business of keeping your properties safe on the internet. We already provide the ability to add OAuth, rate limiting, IP restrictions, and webhook verification to your endpoints. Today, we’re enhancing our security capabilities by providing even more information to protect your endpoints.

All traffic policies now include IP Intelligence, which provides information about the IP addresses hitting your endpoint. This includes geolocation data, autonomous system information, and IP categories, which are collections of IP addresses identified from publicly available information. We make this data available to every request in real-time, enabling you to filter and make informed decisions on managing access to your services with continuous updates from our team.

IP Intelligence is available today for all users leveraging Traffic Policy to manage traffic on their endpoints. Check out our IP Intelligence docs to learn how to add this capability.

Example: How to block Tor traffic from your endpoints

Before we get into the details of what’s available, let’s run through a quick example of how to block Tor traffic to your endpoint. Tor allows users to hide their real identity, which could be undesirable depending on the type of application or API you are hosting.

To block any IP address that is currently being used as a Tor exit node, add this snippet to your Traffic Policy.

---
on_http_request:
  - expressions:
      - "!('proxy.anonymous.tor' in conn.client_ip.categories)"
    actions:
      - type: deny
        config:
          status_code: 403

Since the list of Tor exit nodes is constantly changing as nodes join or leave the network, we automatically update this list of IP addresses so that your endpoint is always protected. 

What data is available in IP Intelligence

All IP addresses in the ngrok network are now automatically tagged with the following information to help you identify and block unwanted traffic from your upstream services.

• Autonomous System Information (conn.client_ip.as.number, conn.client_ip.as.organization): An Autonomous System (AS) is a group of one or more IP addresses run by one or more network operators. All IP addresses have an Autonomous system number (ASN) and organization that identifies the owner of the IP address. For example, one of Amazon’s ASNs is 699 (they have many).
  ‍
• IP Categories (conn.client_ip.categories): A list of tags or labels that allow you to further identify the owner of the IP address or if the IP address has been included on any public blocklists. For example, Datadog runs on Amazon IP addresses, so if you look up a Datadog IP address, the ASN will indicate that this is from Amazon—with IP categories, you can see that this IP is actually controlled by Datadog.

• Geolocation Information (conn.client_ip.geo.location and conn.client_ip.geo.registered_location): All IP addresses also include the geolocation information of where the IP address is located, as well as where it is registered. You can use these to block access to your services from specific countries or regions of the world.

How you can use IP Intelligence today

Like everything at ngrok, we designed our new IP Intelligence system to be easy to use and fit seamlessly into our Traffic Policy engine. When you are building policies for your endpoints, accessing these variables is the same as any other connection or request variable. You can also inject these values into upstream headers for your applications.

---
on_http_request:
  actions:
    - type: add-headers
      config:
        headers:
          x-asn: ${conn.client_ip.as.number}
          x-org: ${conn.client_ip.as.organization}

Behind the scenes, we’ve built a system that fetches a large number of public datasets and aggregates them into a data store that can be dynamically injected into each request. This data comes from lists aggregated from the cybersecurity community, as well as lists published by organizations so that their users can leverage that information when building out integrations with their systems. The full list of sources is available in our IP Categories documentation.

We handle the collection and aggregation of these data sources for you, which reduces the amount of code you need to write and maintain for your service, allowing you to focus on your customers instead of constantly maintaining allow/denylists or actively dealing with unwanted traffic.

Other ways to control (un)wanted traffic

In addition to blocking bad traffic from your endpoints, you can also leverage IP Intelligence to ensure that only specific services with well-defined IP ranges can access your endpoints. The following example ensures that only IP addresses from the AWS Simple Notification Service (SNS) are able to access your endpoint:

---
on_http_request:
  - expressions:
      - "'com.aws.sns' in conn.client_ip.categories"
    actions:
      - ...

You can use this functionality to add additional security to webhook endpoints.

Blocking known bots and scrapers from companies like Censys only takes a single expression as well, since their IPs are included as a category.

---
on_http_request:
  - expressions:
      - "!('com.censys.scanners’ in conn.client_ip.categories)"
    actions:
      - ...

Beyond blocking traffic from known IP lists, you may want to go a step further and block access to your endpoint from specific countries. Doing this is just as easy using the geolocation data included with all client IP addresses.

---
on_http_request:
  - expressions:
      - conn.client_ip.geo.location.country_code == 'RU'
    actions:
      - ...

Observe IP Intelligence with ngrok’s Traffic Inspector

Our Traffic Inspector is a great way to quickly see the requests that are hitting your HTTP endpoints. We’ve added IP Intelligence to every request as well, making it simple to see who’s hitting your endpoint and use that information to update your Traffic Policy rules. 

The impact of IP Intelligence on real-world traffic

At ngrok, we try to use our technology to host our own production services. We recently updated our home page at ngrok.com to be fronted by ngrok, which means it has access to the same IP Intelligence you can leverage. 

We recently mitigated a DDoS attack on ngrok.com where, at its peak, we saw a 93,000% spike in the number of requests to our site. Looking at the traffic hitting our site, we were easily able to get a breakdown of the traffic categories and realized the bulk of traffic was coming from IPs that are known proxies and datacenters. This is a dramatic shift from our usual traffic patterns from residential ISPs.

We were then able to adjust our Traffic Policy rules to look for traffic from these categories and route them somewhere else, like a static page, to help offload some of the load for our main website or block them altogether.

Take advantage of more intelligence today

IP Intelligence gives you more insights into the traffic that is hitting your ngrok endpoints—you can now quickly route traffic based on who owns a specific IP address, where it is located, and if it is connected to any blocklists or specific companies. These lists are dynamically kept up to date by ngrok, eliminating the maintenance burden on your side and streamlining protection for your endpoints.

All you need to get started is an ngrok account and at least one cloud or agent endpoint. Then, be sure to check out our IP Intelligence docs for all the implementation details.

Armed with this automatically updated IP information, you can make informed decisions about how to handle incoming traffic. Whether it’s blocking suspicious IP addresses, prioritizing trusted sources, or implementing specific traffic policies for certain regions or organizations, IP Intelligence empowers you to tailor your endpoint security to meet your unique needs. This level of granular control helps safeguard your services against threats like bots, scrapers, and malicious actors, while ensuring legitimate traffic flows smoothly.

If you have feedback about the new IP Intelligence variables or a data source you would like to see, please let us know so we can improve the system for everyone.