K8s security - Episode 2: Secure your K8s cluster at its root

K8s security - Episode 2: Secure your K8s cluster at its root Deploy • Emmanuelle Demompion • 03/09/21 • 8 min read

We will be starting our journey in the Cloud, securing a Kubernetes cluster and all its layers. Speaking of layers, we like to think that Kubernetes is similar to an onion where one layer of abstraction always hides the next one. Each of these layers can be secured in various ways.

This article is an overview of security objects and practices that everyone should be aware of. Knowing that all these layers of security exist, what they are called, and what they are for, is already a first big step into securing a Kubernetes cluster.

Trying to apply all these concepts at once would be overwhelming and discouraging. Instead, we suggest that you take the information you need for your use cases, and just keep following this series of articles through to the end. Once you have a full overview from the Cloud to your code, it will be easier to come back to this article and sharpen your understanding of these concepts if need be.

The Cluster Overview

Focus personae

The Kubernetes architecture that we describe below has been voluntarily simplified. Our goal is to understand what needs to be secured and, most importantly, how.

What is a Kubernetes cluster made of?

For simplification purposes, we will consider a simple Kubernetes cluster and its main components:

redundant Control Planes

highly available Load Balancing

worker nodes

ETCD key-value store, keeping the state of the cluster at every time

When triggering an action in a cluster kubectl, the user reaches its Control Plane. Users do not access their worker nodes directly, as the cluster services are considered managed. As a result, all handmade instructions on a Kubernetes cluster can potentially be overridden.

Cluster Management, Certificates, and Control Plane Access

focus personae

In terms of security issues, you can either manage your cluster yourself or choose a managed Kubernetes engine. No matter what you decide, each cloud provider can manage cluster security according to their own standards. Bear in mind also that you might have no information about the default security layers installed at cluster creation.

It is highly recommended to be careful and not trust what you don't see. When using a cloud provider, your first step should be to check their default properties, such as Admission Controllers. If you choose to go on a Kubernetes journey by yourself, Admission Controllers are surely one of the first padlocks you should think about.

Worker access

When sending an instruction to a Kubernetes cluster, you are calling an API server, instructing your control plane. Configurations are then applied to the worker nodes.

It is crucial to keep in mind that at any moment, your Kubernetes cluster can auto-heal, replacing or restarting non-responding worker nodes. It can also scale up and down in terms of pods and nodes. This means that Kubernetes worker nodes are stateless and should remain so, at all times.

Accessing a worker node without using the control plane would mean that you apply a configuration manually on a worker node, considering it stateful (which it isn't). It would represent a risk of data loss, configuration loss, or configuration override. Also, you might spend a useless amount of time trying to fix or understand how it happened.

Scaleway's Kubernetes Kapsule does not allow you to access your worker nodes using ssh, specifically so as to avoid this kind of behavior.

If you are building your own Kubernetes cluster and managing it yourself, it is highly recommended that you implement the same behavior, allowing worker nodes access for necessary administration but rejecting it for any non-admin user.

Control plane

At the heart of your cluster, your control plane should be loved, cherished, and protected.

If you are using a managed Kubernetes engine, there is very little probability that you can access it any other way than using the API Server. If you are not using this access method, you must make sure only administrators have access to the servers themselves and that k8s users only have access to the API Server. Each cluster should aim to have its own certificate for accessing it.

When choosing your Cloud Provider for a managed Kubernetes engine, or looking at the best way to manage your certificates yourself, there are a few interesting things to think about while making your decision:

Who needs to access the Control Plane?

Imagine a CI/CD manages your deployments. Do developers need to access the cluster? Usually, the answer is "probably not." Then certificate access can be restricted to the CI/CD tools and administrators managing the cluster.

How to renew the certificate?

If your certificate has leaked for some unknown reason, how can you renew it? Or, if a company administrator leaves, how can you revoke his cluster access?

Cluster security

focus personae

On a Kubernetes cluster level, there are at least three main layers to secure:

Nodes

Network

Traffic routing

Once you have deployed containers, applications, services, and more in your cluster, it is nearly impossible to manage all the security layers.

As all your pods and services might not be deployed simultaneously or with similar configurations, one application could work just fine until a worker node is replaced and your container cannot be pulled back.

Admission controllers

focus personae

Admission controllers control requests accessing the API Server. They will grant or deny access to the cluster workers. They can limit creation, deletion, modification, or even connection.

By setting admission controllers on your cluster, you define the behavior of every Kubernetes object running onto it. It defines security strategies to apply, regardless of pods or containers.

Eight main admission controllers

There are many different admission controllers, and we encourage you to visit the official Kubernetes documentation if you want to go into more detail and find out what other restrictions you can set at cluster level. The eight admission controllers chosen here are easy to understand:

AlwaysPullImages

Setting the default policy of every new pod ensures that your services always use the last version of the container you are pulling. It also implies that only users with permission to do so can fetch containers.

Always pull images policy

DefaultStorageClass

This admission controller…