NVIDIA/cluster-readiness-engine

NVIDIA/cluster-readiness-engine

Description: NVIDIA Cluster Readiness Engine

Language: Go

License: Apache-2.0

Stars: 22

Forks: 6

Open issues: 12

Created: 2026-08-04T12:21:05Z

Pushed: 2026-09-01T21:41:06Z

Default branch: main

Fork: no

Archived: no

README:

NVIDIA Cluster Readiness Engine (NVCRE)


New GPU clusters often contain faulty nodes, and those faults surface only under real distributed load. NVCRE is a Kubernetes controller that certifies GPU clusters before production workloads run on them. It runs real training and communication workloads across topology-aware node groups, measures performance, detects hardware failures, and reports every bad node with a reason. Quarantine is left to your platform: NVCRE never cordons, taints, or otherwise modifies a node.

NVCRE is for platform and infrastructure teams that bring up, validate, or resell GPU clusters.

Features

• A certification catalog with NCCL communication tests and multi-node training workloads
• Platform detection (AWS, GCP, Azure, OCI, nscale, TogetherAI, Mistral, Forge, on-prem) and GPU architecture detection (GB200, GB300, H100, A100, L40S, L40)
• Goodput measurement parsed from training logs with configurable LogProfile patterns
• Per-bus bandwidth measurement parsed from NCCL logs
• Node health monitoring with CEL expressions while workloads run
• Per-node failure reporting with a reason for every failed node
• Topology-aware node grouping and adaptive fault isolation
• Checkpoint restart for training jobs
• WorkloadRun, a single resource to run a training, NCCL, or custom workload
• The nvcrectl CLI for setup, render, run, report, and cleanup

How it works

The APIs compose like Deployment, ReplicaSet, and Pod:

flowchart LR
C[Certification] -->|one per catalog category| W[Workflow]
R[WorkloadRun] -->|one| W
W -->|one| J[Job]
J -->|adapter| T[TrainJob and other workloads]
J -.-> G[GoodputMeasurement]
J -.-> B[BandwidthMeasurement]

A Certification creates one Workflow per catalog category. A WorkloadRun is the single-run entry point: it creates one Workflow directly from its inline workload spec, bypassing the catalog. This is what nvcrectl workloadrun run uses for ad-hoc workloads. Each Workflow creates a Job from its template. The Job creates the workload through an adapter, for example a Kubeflow Trainer TrainJob. Measurement resources parse pod logs with LogProfile regex patterns and compute goodput and bandwidth. When a node fails, NVCRE records it in the certification result with a reason. NVCRE does not modify nodes; quarantine is left to your platform.

Quickstart

For complete prerequisites, installation, a first run, and cleanup, see [Your first certification](docs/getting-started/first-certification.md).

0. Check the cluster

NVCRE requires the NVIDIA GPU Operator and, with the default metrics settings, the Prometheus Operator CRDs that serve monitoring.coreos.com/v1; GB200 and GB300 catalog entries also require the NVIDIA DRA driver for ComputeDomain resources. The diagnostics/dcgm-level4 category additionally requires the standalone DCGM service, which the GPU Operator creates only when spec.dcgm.enabled is true. Because the operator normally uses embedded DCGM for metrics, standalone DCGM is off by default; enable it with:

kubectl patch clusterpolicy cluster-policy --type=merge \
-p '{"spec":{"dcgm":{"enabled":true}}}'

Run kubectl nvcre setup status at any time to see what is present.

1. Install the CLI

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

releases/latest resolves to the newest stable release. To pin a version, download the installer from that release and pass the tag:

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download//installer | bash -s -- -v

The installer places nvcrectl on your $PATH and creates a kubectl-nvcre symlink so the CLI is also available as kubectl nvcre.

2. Set up the cluster

kubectl nvcre setup init

This installs Kubeflow Trainer, the NVCRE CRDs, the controller, and the built-in LogProfiles. The image and chart are public on GHCR; if your cluster pulls from a private mirror instead, pass --image-pull-secret to create the pull secret.

3. Certify

kubectl nvcre certification run \
--category communication/nccl-all-reduce \
--wait

4. Report

The report prints when the run completes. To print it again later, pass the name and the namespace from the run output:

kubectl nvcre certification report -n

╔════════════════════════════════════════════════════════════════╗
║ Certification Report ║
╚════════════════════════════════════════════════════════════════╝

Name: nvcrectl-20260806-162730
Platform: aws
GPU: gb300
Nodes: 16

┌────────────────────────────────────────────────────────────────┐
│ communication/nccl-all-reduce │
├────────────────────────────────────────────────────────────────┤
│ Status: Succeeded │
│ Runtime: 3m 56s │
│ Scale: full-scale │
│ Nodes/Job: 16 │
│ Jobs: 1 │
│ MNNVL: Enabled │
│ │
│ Bandwidth: │
│ Size AlgBW BusBW Samples │
│ 16 GB 473.44 GB/s 932.09 GB/s 9 │
└────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────┐
│ Summary │
├────────────────────────────────────────────────────────────────┤
│ Categories: 1/1 passed │
│ Failed Nodes: none │
│ Result: PASSED │
└────────────────────────────────────────────────────────────────┘

Install from the registry instead

setup init above is the quickest path. If you would rather manage NVCRE with Helm, or need to pin the controller image in your own manifests, both are published to the GitHub Container Registry on every release.

# Resolve the newest stable release (no authentication needed)
NVCRE_VERSION=$(curl -fsSL https://api.github.com/repos/NVIDIA/cluster-readiness-engine/releases/latest | jq -re .tag_name)
: "${NVCRE_VERSION:?no stable release found}"

# Inspect the chart before installing it
helm show chart oci://ghcr.io/nvidia/cluster-readiness-engine --version "$NVCRE_VERSION"

helm install nvcre \
oci://ghcr.io/nvidia/cluster-readiness-engine \
--version "$NVCRE_VERSION" \...