togethercomputer/core-dump-handler

togethercomputer/core-dump-handler

Description: Save core dumps from a Kubernetes Service or RedHat OpenShift to an S3 protocol compatible object store

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2025-09-18T21:15:55Z

Pushed: 2025-09-17T21:21:19Z

Default branch: main

Fork: yes

Parent repository: IBM/core-dump-handler

Archived: no

README:

Core Dump Handler

This helm chart is designed to deploy functionality that automatically saves core dumps from most public cloud kubernetes service providers and private kubernetes instances to an S3 compatible storage service.

  

Contributions

Please read the [CONTRIBUTING.md](CONTRIBUTING.md) it has some important notes. Pay specific attention to the Coding style guidelines and the Developer Certificate of Origin

Code Of Conduct

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

[The full code of conduct is available here](./code-of-conduct.md)

Install

Please refer to the chart README.md for full details.

Kubernetes Service Compatibility

This is a matrix of confirmed test targets. Please PR environments that are also known to work

ProviderProductVersionValidated?Working?

AWSEKS1.21YesYes

AWSROSA4.8YesYes

Custom BuildK8SN/AYesYes

Digital OceanK8S1.21.5-do.0YesYes

GoogleGKE-cos_containerd1.20.10-gke.1600YesYes

GoogleGKE-Ubuntu1.20.10-gke.1600YesYes

IBMIKS1.19-1.21YesYes

IBMROKS4.6-4.8YesYes

MicrosoftAKS1.19YesYes

MicrosoftARO4.8YesYes

RedHatOn-Premises4.8YesYes

Background

Core Dumps are a critical part of observability.

As systems become more distributed core dumps offer teams a non-invasive approach to understanding why programs are malfunctioning in any environment they are deployed to.

Core Dumps are useful in a wide number of scenarios but they are very relevant in the following cases:

• The process exits without a useful stack trace

• The process runs out of memory

• An application doesn’t behave as expected

The traditional problems with core dumps are:

• Overhead of managing the dumps

• Dump Analysis required specific tooling that wasn't readily available on the developers machine.

• Managing Access to the dumps as they can contain sensitive information.

This chart aims to tackle the problems surrounding core dumps by leveraging common platforms (K8s, ROKS and Object Storage) in a cloud environment to pick up the heavy lifting.

Chart Details

The chart deploys two processes:

1. The agent manages the updating of /proc/sys/kernel/* configuration, deploys the composer service and uploads the core dumps zipfile created by the composer to an object storage instance.

2. The composer handles the processing of a core dump and creating runtime, container coredump and image JSON documents from CRICTL and inserting them into a single zip file. The zip file is stored on the local file system of the node for the agent to upload.

When you install the IBM Cloud Core Dump Handler Helm chart, the following Kubernetes resources are deployed into your Kubernetes cluster:

• Namespace: A specific namespace is created to install the components into - defaults to ibm-observe

• Handler Daemonset: The daemonset deploys a [pod](./charts/core-dump-handler/templates/daemonset.yaml) on every worker node in your cluster. The daemonset contains configuration to enable the elevated process to define the core pattern to place the core dump into object storage as well as gather pod information if available.

• Privileged Policy: The daemonset configures the host node so priviledges are required.

• Service Account: Standard Service account to run the daemonset

• Volume Claims: For copying the composer to the host and enabling access to the generated core dumps

• Cluster Role: Created with an event resource and create verb and associated with the service account.

Component Diagram

Service Component Layout


Permissions

To install the Helm chart in your cluster, you must have the Administrator platform role.

Security implications

This chart deploys privileged kubernetes daemonset with the following implications:

1. the automatic creation of privileged container per kubernetes node capable of reading core files querying the crictl for pod info.

2. The daemonset uses hostpath feature interacting with the underlying Linux OS.

3. The composer binary is deployed and ran on the host server

4. Core dumps can contain sensitive runtime data and the storage bucket access must be managed accordingly.

5. Object storage keys are stored as secrets and used as environment variables in the daemonset

Resources Required

The IBM Cloud Core Dump Handler requires the following resources on each worker node to run successfully:

• CPU: 0.2 vCPU
• Memory: 128MB

Updating the Chart

1. Delete the chart. Don't worry this won't impact the data stored in object storage.

$ helm delete core-dump-handler --namespace observe

2. Ensure the persitent volume forhost-name are deleted before continuing

$ kubectl get pvc -n observe

3. Install the chart using the same bucket name as per the first install but tell the chart not to creat it.

$ helm install core-dump-handler . --namespace observe

Removing the Chart

helm delete core-dump-handler -n observe

Build and Deploy a Custom Version

1. Build the image `docker build -t…