May 18, 2022
May 18, 2022
Last week, we fixed a multi-tenancy bug in the ngrok dashboard’s caching layer that unintentionally leaked data between a small subset of ngrok accounts when they viewed the ngrok dashboard.
This bug affected less than 5% of ngrok’s active users. We have contacted all accounts affected by the bug directly via email with instructions for remediation.
If you have not received an email, your account was not affected.
We have neither found evidence nor received reports of any malicious activity related to the bug.
The personal ngrok Authtoken of affected users may have been viewed by another ngrok user. This is the most serious repercussion. An ngrok Authtoken is a scoped credential that has limited capabilities: it cannot be used to modify or access your account’s configuration data. However, it can be used to start new tunnels on your ngrok account.
Account data, including your email address, online tunnels, reserved domains and addresses, and other configuration data may have been viewed by another ngrok user.
The failure was read-only. No affected account could modify another’s.
No traffic across tunnels or endpoints was ever viewable to the other account.
No passwords, API keys, SSH Keys, additional Authtokens, Credit Cards or Payment Data were viewable by another account.
This bug affected less than 5% of ngrok’s active accounts.
The majority of accounts were only affected by a single request leak.
The logs do not identify any malicious patterns of exploitation of the bug.
This was not a security breach by a malicious attacker and ngrok’s systems have not been compromised.
All affected users have been contacted directly via email with the recommendation to rotate their Authtoken using the following steps:
Ngrok’s internal RPC systems perform a type of caching called ‘request coalescing’. For certain annotated methods, if two RPCs occur simultaneously and they specify the same request parameters, the system only performs a single RPC and returns the same result to both callers.
This optimization was applied to the RPC that loads an ngrok account’s dashboard state. Unlike most requests in our system, the request object for this RPC does not include the ID of the account in the request parameters because that data is pulled from the dashboard user’s session state. This meant that requests by two different users for their dashboard state could be incorrectly collapsed into a single request.
For example, two simultaneous calls to GetAccount(id=10) and GetAccount(id=10) could be coalesced but GetAccount(id=9) and GetAccount(id=10) would not. The call to load the ngrok dashboard loads the account ID via session state, so it is expressed as LoadDashboard() and not LoadDashboard(id=10). This resulted in the bug where two simultaneous calls to LoadDashboard() for different accounts could be coalesced.
More plainly, if Alice and Bob both loaded the dashboard at the same time, and Bob's request reached the dashboard service while Alice's request was being processed, they would both receive the response meant for Alice's dashboard, letting Bob view Alice's dashboard state.
On February 2nd 2022, we deployed a change that enabled request coalescing to the ngrok Dashboard that could result in account information from one user account being displayed to both users.
On May 7th, we received a report from a user describing the unexpected behavior of seeing another user’s account information.
ngrok’s engineering team began investigating, eventually building a reproduction of the bug and crafting a fix.
On May 11th, the fix for the bug was deployed to production.
ngrok’s engineering team began performing forensics on log data to assess the impact of the bug across all accounts as well as to review the data for evidence of any malicious activity. We enumerated all instances where one account could have viewed information from another.
On May 17th, the Engineering and Customer Success teams contacted affected users via email with a notice and remediation steps.
We’ve used this situation to start a comprehensive analysis of how we handle and process log data, detect patterns within it, and respond effectively. These lessons will help us detect suspicious activity sooner and at a larger scale than before.
With regards to this specific incident, we are:
Please reach out with any questions to support-may2022@ngrok.com
