Secure Tunneling to Kubernetes services

Using Argo Tunnels to protect services deployed on k8s.

Overview

We’re looking to create a private, egress-only link between a service running on a Kubernetes cluster and Cloudflare. Users will still be able to access the service, but traffic will only be routed through Cloudflare. Once the tunnel is set up, you can go ahead and remove all public ingress from your service - it will only need egress out to Cloudflare.

Prerequisites

You’ll need a Kubernetes cluster with a running service. I followed the EKS Workshop for deploying Kubernetes on AWS, but you can deploy this wherever you’d like. If you’re following the workshop, be aware that there are two sets of prerequisites to make sure you’ve got in place before you deploy your cluster. Then you can go ahead and deploy one of the backend services (you’ll only need one).

Make sure you’ve got cloudflared and kubectl installed somewhere.

Creating Your Tunnel

Wherever you have cloudflared installed, create an Argo Tunnel:

cloudflared tunnel create k8s-tunnel

This command will create a credential file called something like:

/home/username/.cloudflared/08e2f098-3240-234a-07ab-987324cab03a.json

Upload this credential file to your Kubernetes cluster:

kubectl create secret generic tunnel-credentials --from-file=credentials.json=/home/username/.cloudflared/08e2f098-3240-234a-07ab-987324cab03a.json

(Make sure you change the path to point to your actual credentials file.)

Update DNS

Go to the Cloudflare dashboard and go to the DNS tab. Create a CNAME record.

The name will be whatever you want your user-facing URL to be. Let’s say our site is example.com and we want to point our users to k8s-tunnel.example.com.

The Target will be your tunnel ID (08e2f098-3240-234a-07ab-987324cab03a in this example) plus .cfargotunnel.com. So in this example our Target would be 08e2f098-3240-234a-07ab-987324cab03a.cfargotunnel.com.

Deploy Cloudflared

Now we’ll deploy Cloudflared onto our cluster, taking the example from here as a starting point. (See appendix for full code - updated sections are in bold.)

Open up cloudflared.yaml. You’ll need to update a few details in your ConfigMap.

1. Update tunnel: example-tunnel to be the name of the tunnel you created earlier. In our case that’s k8s-tunnel:

tunnel: k8s-tunnel

2. Update hostname: tunnel.example.com to point at the DNS record you created earlier. In our case that’s k8s-tunnel.example.com:

- hostname: k8s-tunnel.example.com

3. Find the name of the service that you want to tunnel to. If you’re following the EKS Workshop example, open up ecsdemo-nodejs/kubernetes/service.yaml and look for these lines:

apiVersion: v1
kind: Service
metadata:
  name: ecsdemo-nodejs

    Update service: http://web-service:80 to point to your service. In our case that’s ecsdemo-nodejs:

service: http://ecsdemo-nodejs:80

Deploy cloudflared.yaml by running:

kubectl apply -f cloudflared.yaml

Point your browser at k8s-tunnel.yourdomain.com and your service should be displayed!

Conclusion

That’s it! We’re done! You can go ahead and remove all the internet ingress from your service, and feel safe in the knowledge that Cloudflare is protecting your resources.

Appendix: Code

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 2 # You could also consider elastic scaling for this deployment
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
      - name: cloudflared
        image: cloudflare/cloudflared:2021.4.0
        args:
        - tunnel
        # Points cloudflared to the config file, which configures what
        # cloudflared will actually do. This file is created by a ConfigMap
        # below.
        - --config
        - /etc/cloudflared/config/config.yaml
        - run
        livenessProbe:
          httpGet:
            # Cloudflared has a /ready endpoint which returns 200 if and only if
            # it has an active connection to the edge.
            path: /ready
            port: 2000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
        volumeMounts:
        - name: config
          mountPath: /etc/cloudflared/config
          readOnly: true
        # Each tunnel has an associated "credentials file" which authorizes machines
        # to run the tunnel. cloudflared will read this file from its local filesystem,
        # and it'll be stored in a k8s secret.
        - name: creds
          mountPath: /etc/cloudflared/creds
          readOnly: true
      volumes:
      - name: creds
        secret:
          # By default, the credentials file will be created under ~/.cloudflared/<tunnel ID>.json
          # when you run `cloudflared tunnel create`. You can move it into a secret by using:
          # ```sh
          # kubectl create secret generic tunnel-credentials \
          # --from-file=credentials.json=/Users/yourusername/.cloudflared/<tunnel ID>.json
          # ```
          secretName: tunnel-credentials
      # Create a config.yaml file from the ConfigMap below.
      - name: config
        configMap:
          name: cloudflared
          items:
          - key: config.yaml
            path: config.yaml
---
# This ConfigMap is just a way to define the cloudflared config.yaml file in k8s.
# It's useful to define it in k8s, rather than as a stand-alone .yaml file, because
# this lets you use various k8s templating solutions (e.g. Helm charts) to
# parameterize your config, instead of just using string literals.
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
data:
  config.yaml: |
    tunnel: k8s-tunnel
    credentials-file: /etc/cloudflared/creds/credentials.json
    metrics: 0.0.0.0:2000
    no-autoupdate: true
    ingress:
    - hostname: k8s-tunnel.example.com
      service: http://ecsdemo-nodejs:80
    - hostname: hello.example.com
      service: hello_world
    - service: http_status:404