Kubernetes ingress to services in a Consul service mesh

This guide walks you through setting up a Consul service mesh on Kubernetes and using the ngrok Kubernetes Operator to provide ingress to your services. The ngrok Kubernetes Operator is the official open-source controller for adding public and secure ingress traffic to your k8s services. Consul is a secure and resilient service mesh that provides service discovery, configuration, and segmentation; Consul Connect provides service-to-service authorization and encryption with mutual TLS. Together, Consul secures communication between services in a cluster while ngrok provides public ingress to those services.

What you’ll need

A remote or local Kubernetes cluster with Consul installed or minikube to set up a demo cluster locally.
An ngrok account.
kubectl and Helm 3.0.0+ installed on your local workstation.
A reserved domain from the ngrok dashboard or API; this guide refers to it as <NGROK_DOMAIN>.

Set up a local Consul service mesh on Kubernetes

This guide needs a Kubernetes cluster running a Consul service mesh with Consul Connect and the sidecar injector enabled. If you already have one, skip to Configure the ngrok Kubernetes Operator. If you don’t, follow HashiCorp’s Get Started with Consul on Kubernetes tutorial to create a cluster and install Consul. For a local cluster, minikube works. Make sure Consul Connect is enabled so the mesh can inject sidecars. Verify Consul is installed and all its pods are healthy before continuing:

kubectl get pods --namespace consul

Configure the ngrok Kubernetes Operator

Consul requires a bit of extra configuration to work with ngrok’s Operator for Kubernetes ingress. You’ll need to use a pod annotation to enable the Consul Connect sidecar injector. This allows using Consul Connect to secure the traffic between the ngrok Kubernetes Operator and your services.

First, create a Kubernetes Service for the ngrok Kubernetes Operator in the consul namespace. Consul relies on this to name services to declare Service Intention source and destination values.

apiVersion: v1
kind: Service
metadata:
  name: ngrok-operator
  namespace: consul
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app.kubernetes.io/name: ngrok-operator

Next, install the ngrok Kubernetes Operator into your cluster. Its pods must join the Consul service mesh so they can proxy traffic to your other services. You do this with two pod annotations: one to enable the Consul Connect sidecar injector and one to allow outbound traffic through the mesh. Consul documents these annotations in Configure Operators for Consul on Kubernetes.

# Enables the Consul Connect sidecar injector
consul.hashicorp.com/connect-inject: "true"
# The CIDR of your Kubernetes API, from: kubectl get svc kubernetes --output jsonpath='{.spec.clusterIP}'
consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: "10.96.0.1/32"

Follow the Kubernetes ingress quickstart to install the Operator with your ngrok credentials. Then set the annotations with a Helm upgrade, replacing the CIDR with your own:

helm upgrade ngrok-operator ngrok/ngrok-operator \
  --namespace ngrok-operator \
  --reuse-values \
  --set-string 'podAnnotations.consul\.hashicorp\.com/connect-inject=true' \
  --set-string 'podAnnotations.consul\.hashicorp\.com/transparent-proxy-exclude-outbound-cidrs=YOUR_KUBERNETES_API_CIDR'

Consul annotation: HashiCorp’s docs also mention the annotation consul.hashicorp.com/transparent-proxy-exclude-inbound-ports. This does not apply to the ngrok Kubernetes Operator, which creates an outbound connection for ingress rather than exposing ports.Helm: The backslashes escape the dots in the annotation keys, and --set-string keeps each value as a string rather than a boolean, which pod annotations require.Production: To manage your credentials with Infrastructure as Code or source control, see the Helm configuration reference.

Install a sample application

Install a sample application to try out the service mesh and Operator combination. This guide uses the HashiCups Demo Application provided by HashiCorp. This application is a simple e-commerce application that allows users to order coffee cups. It has a frontend and public API services that are also backed by a private API and database. These communicate with each other through the Consul service mesh. It comes with nginx installed as a proxy for the frontend and Public API services. Replace this with ngrok to provide public access and other features.

For this demo, everything is installed in the consul namespace.The ngrok Kubernetes Operator can send traffic to services across different namespaces, but Consul Service Intentions across namespaces require an enterprise account. For now, keep everything in the same namespace.

Clone the HashiCorp Learning Consul repo, which includes example applications for Consul and Kubernetes.
git clone https://github.com/hashicorp/learn-consul-kubernetes
Install the HashiCups sample app in the consul namespace. It consists of several Services and Deployments that make up a tiered application.
kubectl apply -f learn-consul-kubernetes/service-mesh/deploy/hashicups -n consul
Replace the existing Service Intentions with new ones. Service Intentions are how Consul configures the mesh to allow or deny traffic between services. HashiCups ships with nginx intentions to the frontend and public-api services. Remove these and add new intentions that allow traffic from the ngrok Kubernetes Operator to the frontend and public-api services.
kubectl delete serviceintentions public-api -n consul kubectl delete serviceintentions frontend -n consul

Create Service Intentions that allow ngrok to reach the frontend and public-api services. Save the following manifests to a file and apply them with kubectl apply -f <file> -n consul.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: ngrok-consul-frontend
  namespace: consul
spec:
  destination:
    name: frontend
  sources:
    - action: allow
      name: ngrok-operator

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: ngrok-consul-api
  namespace: consul
spec:
  sources:
    - name: frontend
      action: allow
    - name: ngrok-operator
      action: allow
  destination:
    name: public-api

Configure Public Ingress for the sample application

Now that the ngrok Kubernetes Operator can communicate with the frontend service and public-api service through the Consul Service Mesh via Service Intentions, create an ingress to route traffic to the app. Create ingress objects to route traffic to the frontend service and the public-api service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-consul
  namespace: consul
spec:
  ingressClassName: ngrok
  rules:
    - host: <NGROK_DOMAIN>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 3000
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: public-api
                port:
                  number: 8080

This ingress object:

Uses the ngrok ingress class
The host is the ngrok domain name you selected that is static
There is a route for / that routes to the frontend service on port 3000
There is a route for /api that routes to the public-api service on port 8080

Open your <NGROK_DOMAIN> domain in your browser to see the HashiCups application.

Add OAuth protection to the app

Now that HashiCups is publicly accessible through ngrok, you can add Google OAuth to restrict access without deploying extra infrastructure. With the Traffic Policy system and the oauth action, ngrok handles OAuth entirely on its network. You don’t need to add services to your cluster or change any routes, because ngrok authenticates and authorizes requests before they reach your endpoint. To enable the oauth action, create an NgrokTrafficPolicy custom resource and apply it to your Ingress with an annotation. You can also apply the policy to a specific backend or as the default backend for an Ingress. See the doc on using the Operator with Ingresses for details. Edit your existing ingress configuration with the following. Note the new annotations field and the NgrokTrafficPolicy CR, which must be in the same namespace as the Ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-consul
  namespace: consul
  annotations:
    k8s.ngrok.com/traffic-policy: oauth
spec:
  ingressClassName: ngrok
  rules:
    - host: <NGROK_DOMAIN>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 3000
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: public-api
                port:
                  number: 8080
---
# Traffic Policy configuration for OAuth
apiVersion: ngrok.k8s.ngrok.com/v1alpha1
kind: NgrokTrafficPolicy
metadata:
  name: oauth
  namespace: consul
spec:
  policy:
    on_http_request:
      - type: oauth
        config:
          provider: google

Re-apply your configuration. When you open your demo app again, ngrok asks you to log in with Google. To limit access to just yourself or your colleagues, tighten the policy. Use expressions and CEL interpolation to reject OAuth logins that aren’t under example.com. Update the NgrokTrafficPolicy portion of your manifest after changing example.com to your domain.

# Traffic Policy configuration for OAuth
apiVersion: ngrok.k8s.ngrok.com/v1alpha1
kind: NgrokTrafficPolicy
metadata:
  name: oauth
  namespace: consul
spec:
  policy:
    on_http_request:
      - type: oauth
        config:
          provider: google
      - expressions:
          - "!actions.ngrok.oauth.identity.email.endsWith('@example.com')"
        actions:
          - type: custom-response
            config:
              body: You are not authorized, ${actions.ngrok.oauth.identity.name}.
              status_code: 400

Check your deployed HashiCups app again. If you log in with an email that isn’t under your domain, ngrok rejects the request.

What’s next?

You’ve used the ngrok Kubernetes Operator to provide public ingress to services inside a Consul service mesh and secured them with OAuth. To go further, explore the Kubernetes docs or try bindings and endpoint pooling.

General Guides

CLI Credentials

Dashboard SSO

Databases

Endpoints SSO

Event Destinations

Frameworks

JWT Validation