MFA for your production infrastructure running on ngrok | ngrok blog
October 18, 2023
The Security and Identity (SI) team at ngrok recently launched support for MFA as an option to secure access to your account’s dashboard. In this post we’re excited to share that journey, from the business reasoning for the project to the technical and security considerations we encountered along the way.
Production infrastructure requires a higher bar for account security
ngrok was built to bring secure connectivity to apps in test/dev so that developers can easily simulate real-world environments. Its primary uses were to enable ingress to developer environments behind NAT or firewalls. There was nothing explicitly barring you from using it for ingress to production services. But most production deployments have a publicly-reachable IP address. As such, there were often few “push” factors making you reach for such an ingress solution in production.
However, with features added in the last two years, ngrok has begun targeting a broader class of ingress-related challenges. Implementing systems for oauth integration and webhook verification are important aspects of ingress. For most products, though, they are table stakes—not the core capability of the application. They are barriers to entry instead of differentiating features for your product. Allowing ngrok's cloud ingress to manage these concerns is now a strong "pull" factor for engineering teams who want to focus on their product.
Since many customers leverage ngrok to serve production workloads, securing access to your ngrok account is more important than ever. The tools we provide to secure your ngrok account must then be correspondingly powerful.
What is Multi-Factor Authentication (MFA)?
Multi-factor authentication is a commonly-seen suite of strategies to authenticate a user by something more than just a password. Among the most common of options is “Time-based One-Time Passwords” - TOTPs for short. TOTPs are six (or more)-digit codes that change roughly every thirty seconds. Setup is typically easy - the user scans a QR code that contains a shared secret and some metadata into an authenticator app.
Once the user scans the QR code into the authenticator app, it begins displaying a code the user can enter to securely login to the application protected by MFA. The code expires every thirty seconds, and a new one appears.
The application protected by MFA will also provide you with a set of recovery codes—randomly-generated alphanumeric strings that you should store durably. This way, if you lose your authenticator app, you can still recover access to your account, provided the codes were properly stored. (More on this later!)
MFA in a service-oriented architecture
ngrok operates as a service-oriented architecture. Internally, the “ngrok” service is actually many smaller services all of which provide solutions to specific sub-problems. Externally-facing services implement their functionality by calling into one or more of these internal services.
The first step in implementing MFA at ngrok was to add a service to manage and store TOTP devices. We designed it to be generic to the concept of “an additional factor of authentication”, and we made TOTP our first concrete use of it.
When it comes to security, the details are important. Rather than writing and maintaining a bespoke implementation of the TOTP algorithm, we used the pquerna/otp golang library to handle the generation and verification of codes. This was a great pick - the library is incredibly concise and we encountered no surprises. To securely store the TOTP secrets, we relied on a preexisting internal library for encrypting secrets in postgres.
We then modified our authentication middleware to make GRPC calls to this service to manage and verify TOTP devices and codes. This helped separate the details of securely storing and verifying TOTP secrets from the concerns of using them to authenticate the user.
MFA for multi-tenant infrastructure
One interesting challenge we faced was that ngrok is “multi-tenant” in the sense that an account is independent of a user. (An account might represent an organization or company, whereas a user represents a human logging in to the service). So a user might be a member of a personal account, an account for a noncommercial project they contribute to, and an account their employer uses. This isn’t too unusual - many IaaS (Infrastructure as a Service) companies have very similar concepts (DigitalOcean, Github, and so on).
This increased the complexity of the problem that our solution needed to engage with. Making an authentication decision had to reason about whether any of the user’s accounts required MFA in addition to whether the user had a device configured.
Security and UX are often opposing forces
A perennial issue for security work is that it often means introducing some small additional user friction in return for (ideally!) greater security benefits. MFA was no exception. We made the decision to make MFA opt-in, so that if users didn’t wish to store recovery keys or manage an authenticator app, they could simply opt out.
A more complicated question was that of how to handle a user who has lost their authenticator app and misplaced their recovery codes. The industry offers much prior art in this matter. A common low-touch method is to simply fall-back to using SMS to send a TOTP code to a phone number on the account. This is very convenient - people rarely lose or change their phone numbers. But it is infamous for reducing the otherwise prodigious security offered by MFA. It is generally infeasible to steal a TOTP secret from an authenticator app. This is in contrast to SMS-based MFA which is vulnerable to the altogether-too-easy SIM hijacking attack. In SIM hijacking, attackers social-engineer a phone provider into issuing a new SIM card to them for your phone number. They then use this access to receive your text messages to bypass MFA and access a target’s account.
Alternatively, a common high-touch method is to let support handle verifying the user. Support will typically use billing details or verify a name on the account against a presented government ID. For ngrok, we chose the support method because of its stronger security posture.
Go forth and generate TOTP codes
You can now use MFA to secure your and your organization’s ngrok accounts! This was a smashing project for the SI team, and we enjoyed building such an impactful feature for our customers - safeguarding your production infrastructure is a key priority for both our team and ngrok as an engineering organization.
Now that you know how to secure your ngrok account, we’ve got you covered with more awesome security related content on the ngrok blog: