coreweave/kubemod

coreweave/kubemod

Description: Universal Kubernetes mutating operator

Language: Go

License: BSD-3-Clause

Stars: 0

Forks: 0

Open issues: 0

Created: 2021-04-13T18:24:33Z

Pushed: 2021-04-19T19:16:13Z

Default branch: master

Fork: yes

Parent repository: kubemod/kubemod

Archived: yes

README: [![Build Status][ci-img]][ci] [![Go Report Card][goreport-img]][goreport] [![Docker Image][docker-img]][docker]

KubeMod

KubeMod is a universal Kubernetes mutating operator.

It introduces ModRule — a custom Kubernetes resource which allows you to intercept the deployment of any Kubernetes object and apply targeted modifications to it or reject it before it is deployed to the cluster.

Use KubeMod to:

• Customize opaque Helm charts and Kubernetes operators.
• Build a system of policy rules to reject misbehaving resources.
• Develop your own sidecar container injections - no coding required.

Table of contents

• [Installation](#installation)
• [Deploying our first ModRule](#deploying-our-first-modrule)
• [Common use cases](#common-use-cases)
• [Modification of behavior](#modification-of-behavior)
• [Modification of metadata](#modification-of-metadata)
• [Sidecar injection](#sidecar-injection)
• [Resource rejection](#resource-rejection)
• [Anatomy of a ModRule](#anatomy-of-a-modrule)
• [Match section](#match-section)
• [Patch section](#patch-section)
• [Miscellaneous](#miscellaneous)
• [Namespaced and cluster-wide resources](#namespaced-and-cluster-wide-resources)
• [Target resources](#target-resources)
• [Note on idempotency](#note-on-idempotency)
• [Debugging ModRules](#debugging-modrules)
• [Declarative kubectl apply](#declarative-kubectl-apply)
• [Gotchas](#gotchas)

---

Installation

As a Kubernetes operator, KubeMod is deployed into its own namespace — kubemod-system. Run the following commands to deploy KubeMod.

# Make KubeMod ignore Kubernetes' system namespace.
kubectl label namespace kube-system admission.kubemod.io/ignore=true --overwrite
# Deploy KubeMod.
kubectl apply -f https://raw.githubusercontent.com/kubemod/kubemod/v0.12.0/bundle.yaml

By default KubeMod allows you to target a limited set of high-level resource types, such as deployments and services.

See [target resources](#target-resources) for the full list as well as instructions on how to expand or limit it.

Upgrade

If you are upgrading from a previous version of KubeMod, run the following:

# Delete the KubeMod certificate generation job in case KubeMod has already been installed.
kubectl delete job -l job-name -n kubemod-system
# Make KubeMod ignore Kubernetes' system namespace.
kubectl label namespace kube-system admission.kubemod.io/ignore=true --overwrite
# Upgrade KubeMod operator.
kubectl apply -f https://raw.githubusercontent.com/kubemod/kubemod/v0.12.0/bundle.yaml

Uninstall

To uninstall KubeMod and all its resources, run:

kubectl delete -f https://raw.githubusercontent.com/kubemod/kubemod/v0.12.0/bundle.yaml

Note: Uninstalling KubeMod will also remove all your ModRules deployed to all Kubernetes namespaces.

Deploying our first ModRule

Once KubeMod is installed, you can deploy ModRules to intercept the creation and update of specific resources and perform modifications on them.

For example, here's a ModRule which intercepts the creation of Deployment resources whose app labels equal nginx and include at least one container of nginx version 1.14.*.

The ModRule patches the matching Deployments on-the-fly to enforce a specific securityContext and add annotation my-annotation.

Since KubeMod intercepts and patches resources before they are deployed to Kubernetes, we are able to patch read-only fields such as securityContext without the need to drop and recreate existing resources.

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
name: my-modrule
spec:
type: Patch

match:
# Match deployments ...
- select: '$.kind'
matchValue: Deployment

# ... with label app = nginx ...
- select: '$.metadata.labels.app'
matchValue: 'nginx'

# ... and at least one container whose image matches nginx:1.14.* ...
- select: '$.spec.template.spec.containers[*].image'
matchRegex: 'nginx:1\.14\..*'

# ... but has no explicit runAsNonRoot security context.
# Note: "negate: true" flips the meaning of the match.
- select: '$.spec.template.spec.securityContext.runAsNonRoot == true'
negate: true

patch:
# Add custom annotation.
- op: add
path: /metadata/annotations/my-annotation
value: whatever

# Enforce non-root securityContext and make nginx run as user 101.
- op: add
path: /spec/template/spec/securityContext
value: |-
fsGroup: 101
runAsGroup: 101
runAsUser: 101
runAsNonRoot: true

Save the above ModRule to file my-modrule.yaml and deploy it to the default namespace of your Kubernetes cluster:

kubectl apply -f my-modrule.yaml

After the ModRule is created, the creation of any nginx Kubernetes Deployment resource in the same namespace will be intercepted by KubeMod, and if the Deployment resource matches the ModRule's match section, the resource will be patched with the collection of patch operations.

To list all ModRules deployed to a namespace, run the following:

kubectl get modrules

Common use cases

The development of KubeMod was motivated by the proliferation of Kubernetes Operators and Helm charts which are sometimes opaque to customizations and lead to runtime issues.

For example, consider these issues:

• https://github.com/elastic/cloud-on-k8s/issues/2328
• https://github.com/jaegertracing/jaeger-operator/issues/1096

Oftentimes these issues are showstoppers that render the chart/operator impossible to use for certain use cases.

With the help of KubeMod we can make those charts and operators work for us. Just deploy a ModRule which targets the problematic primitive resource and patch it on the fly at the time it is created.

See the following sections for a number of typical use cases for KubeMod.

Modification of behavior

Here's a typical black-box operator issue which can be fixed with KubeMod: https://github.com/elastic/cloud-on-k8s/issues/2328.

The issue is that when the [Elastic…