nebius/soperator 1.22.0

1.22.0

Repository: nebius/soperator

Tag: 1.22.0

Published: 2025-09-24T11:53:30Z

Prerelease: no

Release notes: Changes made since version 1.21.9 prior to version 1.22.0:

🚀 Features

• [ha controller] add placeholder and single replicas
• PR: #1344
• Bump container toolkit and do not install it into the jail
• PR: #1347
• Bump slurm 24.11.6
• PR: #1352
• Improve job metrics handling in exporter
• PR: #1358
• Speedup populate_jail + don't overwrite existing data
• PR: #1278
• Implement async metrics collection for SLURM exporter
• PR: #1376
• add scrape node conditions and pod termination reasons metrics from ksm
• PR: #1379
• add gpu-fryer check
• PR: #1375
• Add memory latency and bandwidth Active Checks
• PR: #1368
• Add self-monitoring metrics on separate port 8081
• PR: #1382
• add cuda-samples test
• PR: #1381
• add health-checker upgrade
• PR: #1384
• [slurmcluster] maxUnavalaible for workes and support pre install images
• PR: #1391
• add comment reaction on the check
• PR: #1396
• [CD] Support for VMAgent external labels
• PR: #1403
• Add JailedConfig CR
• PR: #1287
• Undrain nodes after user problems + rewrite passive checks on Python
• PR: #1392
• Collecting health_checker_cmd_stdout logs
• PR: #1410
• Standalone Slurm exporter - No Kubernetes needed
• PR: #1415
• [SlurmCluster CRD] rewrite logic of printcolumn
• PR: #1434
• Clean old logs from /opt/soperator-outputs
• PR: #1431
• Add slurm scripts for managing per-job tmpfs directories
• PR: #1432
• move IB checks to health-checker
• PR: #1438
• add aggregation to the JailedConfig
• PR: #1435
• Log one-line JSON outputs for health-checker + rewrite in Python
• PR: #1447
• Chessprofessor/each worker jobs
• PR: #1443
• #1312 Add ib-write-bw/lat cpu checks
• PR: #1323
• add activechecks PrinterColumns
• PR: #1497
• add priorityclassname for components of clurm cluster
• PR: #1519
• Bump health checker 1.0.0-150
• PR: #1526
• Slurm scripts drop cache shmem
• PR: #1554
• remove appormore deny for libEGL_ for running docker active check image
• PR: #1574
• bump nc-health-checker_1.0.0-151.250904
• PR: #1573
• [slurm login] add SshdServiceLoadBalancerSourceRanges to login node
• PR: #1581
• Fix passive check filtering
• PR: #1582

🐛 Fixes

• [nccl-debug] Use chmod instead of umask
• PR: #1345
• remove size from controller spec
• PR: #1356
• fix cache-sync-timeout to k8s default
• PR: #1367
• Don't plan eachWorkerJobArray active checks on bad nodes
• PR: #1378
• remove validation tool init container #1361
• PR: #1372
• [nodeTopology] re-generate CM node topology if cm deleted.
• PR: #1383
• SLURM exporter: use env configuration and improve docs
• PR: #1394
• [worker topology controller] initial cm topology until asts not found
• PR: #1401
• removing deprication fileds
• PR: #1407
• fix priorityclass name for controller
• PR: #1411
• fix issue with metadata.resourceVersion: Invalid value: 0x0: must be specified for an Patch in ASTS
• PR: #1420
• set default values as a defalt in CRD #1421
• PR: #1422
• Pre release fixes 1.22/0
• PR: #1423
• Fix pagination issue in cache
• PR: #1433
• [sopertochecs] change drain reason for maintenance
• PR: #1430
• Disable acctg by default
• PR: #1442
• add aggregation to the JailedConfig
• PR: #1435
• Fix error with read-only workdir in dcgmi_diag_r1 health-checker
• PR: #1450
• [sconfigcontroller] fix reconcile jailedconfig
• PR: #1455
• Fallback on unix.renameat2 to os.rename when renameat2 is not supported
• PR: #1452
• fix getting error when [user_problem] reason #1468
• PR: #1474
• enable nodeLogs by default
• PR: #1477
• fix preemptionPolicy for controller
• PR: #1495
• fix patch cm with empty labels
• PR: #1508
• add resources values for spo
• PR: #1513
• Move Healthchekcer parts to optimize for build time
• PR: #1532
• Cherry-pick active checks in Helm
• PR: #1529
• [nccld-plugin] Make user responsible for correct rights of the output directory
• PR: #1541
• Add fixes for activechecks in 1.22
• PR: #1542
• Slurm scripts drop cache shmem
• PR: #1554
• DCGM Exporter fix for toolkit validation
• PR: #1583
• Add wait after users creation + Split all-reduce-perf + Fix dcgmi_diag_r1
• PR: #1572
• [TopologyController] add EnsureWorkerTopologyConfigMap to check existing of JailedConfig
• PR: #1580
• Fix passive check filtering
• PR: #1582
• change k8up-cleanup image
• PR: #1597
• Extract Enroot's config paths to config dir
• PR: #1599

📦 Dependencies

• Bump docker/login-action from 3.4.0 to 3.5.0
• PR: #1370
• Bump github.com/getkin/kin-openapi from 0.122.0 to 0.131.0
• PR: #1387
• Bump golang.org/x/oauth2 from 0.24.0 to 0.27.0
• PR: #1386
• Bump golang.org/x/net from 0.37.0 to 0.38.0
• PR: #1388
• Bump sigs.k8s.io/yaml from 1.4.0 to 1.6.0
• PR: #1338
• Bump actions/checkout from 4.2.2 to 5.0.0
• PR: #1408
• Bump actions/download-artifact from 4 to 5
• PR: #1389

📔Docs

• SLURM exporter: use env configuration and improve docs
• PR: #1394

Other

• Merge dev to main
• PR: #1332
• Set sconfigcontroller UID and GID from Helm values
• PR: #1327
• DCGM exporter Helm chart: support metricRelablings
• PR: #1425
• [nccl-debug] Don't try chmodding files and directories without a need
• PR: #1522
• Support passing extraArgs for node-exporter
• PR: #1534
• Add reason label to slurm_node_info metric for better observability
• PR: #1538
• [cherry-pick] Add node unavailability and draining duration metrics
• PR: #1561
• Support DCGM Exporter on driverful
• PR: #1578
• Change DCGM exporter image version
• PR: #1595

Contributors: @theyoprst, @mcheshkov, @dstaroff, @Uburro, @asteny, @itechdima, @rdjjke, @ChessProfessor, @dependabot[bot], @ali-sattari, @mateusclira-nv, @dnugmanov, @github-actions[bot]

| 📁 Categorized PRs | 📂 Uncategorized PRs | 📥 Commits | ➕ Lines added | ➖ Lines deleted | | :---: | :---: | :---: | :---: | :---: | | 5613 | 593 | 250 | 9661 | 5211 |