Ingress to services in a vcluster on Kubernetes

TL;DR

To use the ngrok Kubernetes Operator with vcluster in a local cluster:

1. Set up a local virtual cluster
2. Install the ngrok Kubernetes Operator
3. Install a sample application
4. Add OAuth protection to your demo app

The ngrok Operator for Kubernetes is the official controller for adding public and secure ingress traffic to your k8s services. This open source Operator works with any cloud, locally-hosted, or on-premises Kubernetes cluster to provide ingress to your applications, APIs, or other services while also offloading network ingress and middleware execution to ngrok's platform.

vcluster is an open source project for creating virtual clusters that run inside regular namespaces, which provides strong isolation and easy access for multiple tenants with low cost and overhead. The pods you deploy on a vcluster are scheduled inside of the underlying cluster, while other resources, like deployments and CRDs, exist only inside the virtual cluster.

Together, the ngrok Kubernetes Operator and vcluster work to provide secure and load-balanced ingress for services running on a virtual cluster, which lets you isolate development environments, create an internal developer platform (IDP) in cloud native environments, and run experiments or simulations virtually while properly routing external traffic.

With this guide, you'll use an existing Kubernetes cluster, or set up a local development cluster with minikube, to launch a virtual cluster, and deploy a demo application. You'll then deploy the ngrok Kubernetes Operator to connect your demo application to the ngrok platform to route traffic to your vcluster.

This tutorial requires:

1. An ngrok account.
2. The vcluster CLI installed locally.
3. kubectl installed locally.
4. Helm 3.0.0+ installed locally.
5. An existing remote or local Kubernetes cluster OR minikube to create a new demo cluster locally.

Step 1: Set up a local virtual cluster with vcluster

To follow along with this guide, you need a remote or local Kubernetes cluster with vcluster installed. If you have an existing cluster with vcluster set up, you can skip this step and proceed to Step 2: Install the ngrok Ingress Controller.

If you don't have a cluster already, create one locally with minikube and install vcluster.

1. Create a local Kubernetes cluster with minikube.

   Loading…

2. Use the minikube CLI to ensure your new local cluster is running properly.

   Loading…

3. Create a new vcluster with the name my-vcluster, which creates a new namespace called vcluster-my-cluster and automatically switches the active kube context to use your new vcluster.

   Loading…

4. To ensure your new local cluster is running properly get the namespaces for your instance. Your list of namespaces in the my-vcluster context should look something like this.

   Loading…

   If you are not connected to your new vcluster and its kube context, you can run vcluster connect my-vcluster to try again.

   You know have a vcluster installed on your local minikube cluster.

   note

   These steps are partially based the guide Using the ngrok Kubernetes Operator to create Preview Environments with vcluster from Loft, the maintainers of vcluster.

Step 2: Install the ngrok Kubernetes Operator

Now that you have a Kubernetes cluster integrated with vcluster, you can install the ngrok Kubernetes Ingress Controller to provide ingress to any services you want to run on your virtual cluster.

1. Add the ngrok Helm repository if you haven't already.

   Loading…

2. Set up the AUTHTOKEN and API_KEY exports, which allows Helm to install the Operator using your ngrok credentials. Find your AUTHTOKEN under Your Authtoken in the ngrok dashboard.

   To create a new API key, navigate to the API section of the ngrok dashboard, click the New API Key button, change the description or owner, and click the Add API Key button. Copy the API key token shown in the modal window before closing it, as the ngrok dashboard will not show you the token again.

   Loading…

3. Install the ngrok Kubernetes Operator with Helm under a new ngrok-ingress-controller namespace.

   Loading…

4. Verify you have installed the ngrok Kubernetes Operator successfully and that pods are healthy.

   Loading…

Step 3: Install a sample application

At this point, you have a functional vcluster with the ngrok Kubernetes Operator running and authenticated with your ngrok credentials. To demonstrate how the Operator simplifies routing external traffic to your primary cluster, virtual cluster, and ultimately an exposed service or endpoint, you can install a sample application.

1. Create a ngrok static subdomain for ingress if you don't have one already. Navigate to the Domains section of the ngrok dashboard and click Create Domain or New Domain. This static subdomain will be your NGROK_DOMAIN for the remainder of this guide.

   By creating a subdomain on the ngrok network, you provide a public route to accept HTTP, HTTPS, and TLS traffic.

2. Create a new Kubernetes manifest (2048.yaml) with the below contents. This manifest defines the 2048 application service and deployment, then configures the ngrok Kubernetes Operator to connect the game-2048 service to the ngrok edge via your NGROK_DOMAIN.

   tip

   Make sure you edit line 45 of the manifest below, which contains the NGROK_DOMAIN variable, with the ngrok subdomain you created in the previous step. It should look something like one-two-three.ngrok.app.

   Loading…

3. Apply the 2048.yaml manifest to your vcluster.

   Loading…

   tip

   Note: If you get an error when applying the manifest, double check that you've updated the NGROK_DOMAIN value and try again.

4. Confirm your vcluster successfully deployed your 2048 application by navigating to the Edges section in the ngrok dashboard.

   An edge connects your 2048 application, running in your vcluster, to the rest of the world through the ngrok Ingress Controller. You should see a new edge configuration created by kubernetes-ingress-controller that matches the domain name you entered on L45 of your 2048.yaml manifest.

   Under the Backend section of your edge configuration, you can also see cluster details as annotations, like the namespace and connected service, to validate that your demo application is accessible to external traffic via the Operator.

   

   Click on your edge configuration to see additional details and options for advanced ingress needs, like creating multiple tunnels for load balancing, enabling the Mutual TLS module, adding compression, and more. We'll use one of these options, OAuth, in the next step.

5. Access your 2048 demo app by navigating to the your domain, e.g. https://one-two-three.ngrok.app. ngrok's edge and your Operator will route traffic to your app from any device or external network as long as your vcluster remains operational.

   

Step 4: Add OAuth protection to your demo app

Let's take your ingress needs a little further by assuming you want to add edge security, in the form of Google OAuth, to the endpoint where your 2048 application is humming along.

By default, ngrok manages OAuth protection entirely at the cloud edge, which means you don't need to add any additional services to your cluster, or alter routes, to ensure ngrok's edge authenticates and authorizes all requests before allowing ingress and access to your endpoint.

1. Edit your existing 2048.yaml manifest with the following configuration.

   tip

   • See L42-43 for the additional annotation that this OAuth setup requires.
   • See L58-69 for the newly added configuration for ngrok's OAuth module, replacing acme.com or ngrok.com with the domain name for your email address. You can also configure the OAuth module to authenticate individual email addresses.

   Loading…

2. Re-apply your 2048.yaml configuration.

   Loading…

3. Visit the Edges section of the ngrok dashboard, click on the edge for your deployment, then click OAuth to confirm the module is using the configuration you added to your 2048.yaml manifest.

   

4. Access your 2048 app at your domain (e.g. https://one-two-three.ngrok.app) to verify that it requests authentication via Google OAuth before allowing you to access the 2048 app.

What's next?

You've now used the open source ngrok Kubernetes Operator to add secure access to your endpoint without worrying about IPs, network interfaces, or VPC routing. Because ngrok offloads ingress and middleware execution to its global edge, you can follow the same procedure listed above for any Kubernetes environment, like EKS, GKE, and OpenShift, with similar results.

If you want to clean up the work you did for this demo application, the easiest way (and the advantage of virtual clusters in the first place) is to disconnect from your vcluster and then delete it with the vcluster CLI. That will remove the namespace and all its resources, returning your primary cluster to its initial state.

Loading…

You can also extend your new vcluster and ngrok Kubernetes Operator with additional modules, a fanout configuration to route traffic to multiple services from a single domain or IP address, or even multiple Ingress Controller installations running alongside one another.

Learn more about the ngrok Kubernetes Operator, or contribute, by checking out the GitHub repository and the project-specific documentation.