Our general philosophy for keeping our production environments secure has two main components: defense in depth and principle of least privilege.
We practice 'least privilege' access grants. Engineers get the minimum level of production access they need.
Shell access to production machines uses industry best practices of SSH certificate authorities to grant time limited access in extraneous circumstances.
We keep audit logs of all grants to access production machines.
Services that manipulate cloud resources are granted least privilege access grants via an associated 'Role' they assume to perform those operations.
All data is encrypted at rest. This includes databases, host filesystems, network mounted file systems, and data sent to data warehousing services.
All secrets and keys uploaded by users are further encrypted at the application layer with keys that only we control.
All internal secrets used by ngrok are stored encrypted at rest with key rotation using industry secret key storage provided by Hashicorp Vault.
For API keys, credential tokens, and passwords, we only keep one way salted hashes of users' credential tokens.