Best practices on service exposure and data persistence for a Multi-Cloud Kubernetes cluster

Best practices on service exposure and data persistence for a Multi-Cloud Kubernetes cluster Scale • Emmanuelle Demompion • 18/03/22 • 22 min read

This article follows the first and second part of the Hands-On prepared for Devoxx Poland 2021 .

How to deploy and distribute the workload on a multi-cloud Kubernetes environment

⚠️ Warning reminder:

This article will balance between concept explanations and operations or commands that need to be performed by the reader.

If this icon 🔥 is present before an image, a command, or a file, you are required to perform an action.

So remember, when 🔥 is on, so are you!

Load Balancing

Load Balancing traditionally allows service exposure using standard protocols such as HTTP or HTTPS. Often used to give an external access point to software end-users, Load Balancers are usually managed by Cloud providers.

When working with Kubernetes, a specific Kubernetes component manages the creation, configuration, and lifecycle of Load Balancers within a cluster.

Cloud Controller Manager (CCM) component

The Cloud Controller Manager is the Kubernetes component that provides an interface for Cloud providers to manage Cloud resources from a cluster's configuration.

This component is especially in charge of the creation and deletion of Instances (in auto-healing and auto-scaling usage) and Load Balancers to expose the applications running in a Kubernetes cluster outside of it, making it available for external users.

Classic Kubernetes architecture with the CCM component within the Kubernetes Control-Plane

Classic Kubernetes architecture with the CCM component within the Kubernetes Control-Plane

🔥 Adding a Load Balancer

We will create a Multi-Cloud Load Balancer with Scaleway's Cloud Controller Manager which allows us to expose services and redirect traffic across both of our two providers (Scaleway and Hetzner).

For this exercise, we will be using an open source project called WhoAmI hosted on Github . The two main advantages are that the project is available as a docker image, and that once called, it outputs the identifier of the pod it runs on.

🔥 We are going to write a lb.yaml file which will contain:

the deployment of our whoami application, with two replicas , a pod anti affinity to distribute our two pods on different providers, and expose its port 8000 inside the cluster.

the service of type LoadBalancer which maps the 8000 port of our pods to the standard HTTP port ( 80 ) and expose our application outside of the cluster.

🔥

#lb.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: whoami labels: app: whoami spec: replicas: 2 selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - whoami topologyKey: provider containers: - name: whoami image: jwilder/whoami ports: - containerPort: 8000 --- apiVersion: v1 kind: Service metadata: name: scw -multicloud-lb labels: app: whoami spec: selector: app: whoami ports: - port: 80 targetPort: 8000 type: LoadBalancer CopyContentIcon Copy code Now that our yaml file is ready, we can create the deployment and the load balancer service at once.

🔥 kubectl apply -f lb.yaml

Output

deployment.apps/whoami created

service/scw-multicloud-lb created

🔥 We can list the services available in our cluster and see that our service of type LoadBalancer is ready and has been given an external-ip .

🔥 kubectl get services

Output

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) kubernetes ClusterIP 10.32.0.1 443/TCP scw -multicloud-lb LoadBalancer 10.35.41.56 51.159.11.31 80:31302/TCP CopyContentIcon Copy code As we stated before, creating a Load Balancer service within a Kubernetes cluster results in the creation of a Load Balancer resource on the user account on the Cloud provider managing the Kubernetes cluster (via the CCM component).

If we connect to our Scaleway Console, we can see that a Load Balancer resource has been added to our account with the same attributed external-ip that we were given by listing our Kubernetes services .

Load Balancer listing view in Scaleway Console

Our Load Balancer overview in Scaleway Console

Since we created our Load Balancer service in the context of a Kubernetes Kosmos cluster (i.e. a Multi-Cloud environment), the CCM created a Multi-Cloud Scaleway Load Balancer to match our needs.

Checking the behavior of our Load Balancer

The project we deployed allows us to test the behavior of our Multi-Cloud Load Balancer. We already know that our two whoami pods run on different providers, but let's be sure that the traffic is redirected to both providers when it is called.

🔥 To do so, we will create a simple bash command to call our given external-ip . The whoami project should answer us its identifier.

🔥 while true; do curl http://51.159.11.31; sleep 1; done

Output

I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-kbfq4 I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-hxdwp I'm whoami-59c74f5cf4-kbfq4 CopyContentIcon Copy code

When calling our service , we can see that we indeed have two different identifiers returned by our calls, those identifiers corresponding to our two pods running on nodes from our two different providers.

The Multi-Cloud Load Balancer created directly within our Kubernetes cluster can redirect traffic between different nodes from different Cloud providers for the same application (i.e. deployment ).

If we want to verify this assessment, we can simply list our pods again and the nodes they run on.

🔥 kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

Output

NAME NODE STATUS whomai-59c74f5cf4-hxdwp scw -kosmos-kosmos- scw -0937 Running whomai-59c74f5cf4-kbfq4 scw -kosmos-worldwide-b2db Running CopyContentIcon Copy code The last check we can do is to display the provider label associated with these two nodes, just to be sure they are on different Cloud providers.

🔥 kubectl get nodes scw-kosmos-kosmos-scw-09371579edf54552b0187a95 scw-kosmos-worldwide-b2db708b0c474decb7447e0d6 -o custom-columns=NAME:.metadata.name,PROVIDER:.metadata.labels.provider

Output

NAME PROVIDER scw -kosmos-kosmos- scw -09371579edf54552b0187a95 scaleway scw -kosmos-worldwide-b2db708b0c474decb7447e0d6 hetzner…