NVIDIA/cloudai v1.4.0

v1.4.0

Repository: NVIDIA/cloudai

Tag: v1.4.0

Published: 2025-10-28T13:38:27Z

Prerelease: no

Release notes:

Highlights

1. GB300 Support for Common Configs

CloudAI and example configurations now support GB300 systems, expanding hardware compatibility.

2. AI Dynamo with Kubernetes Support (Alpha)

• Added Kubernetes SPCx support for AI Dynamo workload
• Enhanced container orchestration capabilities

3. New Model and Workload Support

• Qwen Recipe Support: Added comprehensive Qwen model recipe support
• SGLang Backend: Added SGLang backend support for DeepSeekR1 model
• NIXL KVBench: New NIXL KVBench workload with full Slurm integration

4. Advanced AI Dynamo Capabilities

• Multi-worker GPU Slicing: Multi-worker-per-node GPU slicing with dynamic allocation
• Explicit Node Assignment: Dedicated node assignment for prefill and decode workers
• Shell Script Entry Point: Shell script-based entry point (replacing Python implementation)
• Environment Validation: Environment validation during startup sequence
• Error Handling: Error detection and retry mechanism for worker failures

5. Agent System Enhancements

• Plugin System: Load agents from entrypoints for extensibility
• Custom Reward Functions: Custom reward functions with latency & throughput metrics handling
• Configurable Rewards: Configurable agent reward functions

6. Enhanced Documentation

• Sphinx Framework: Sphinx-based documentation framework with GitHub Pages deployment
• Comprehensive Workload Docs: Complete documentation for Bash, NCCL, UCC, and other workloads
• Interactive Features: Copy-to-clipboard support for code snippets

7. CLI Modernization

• Click Framework: Complete CLI re-implementation using Click framework (replacing argparse)
• Improved Usability: Made --tests-dir optional for better user experience
• Simplified Commands: Removed unnecessary install/uninstall command options

8. NIXL Workload Improvements

• Per-rank Environment Variables: Enhanced per-rank environment variable support for NIXL Perftest
• Multi-backend Support: Support for non-UCX backends with multiple backend options
• Enhanced Output Parsing: Improved output parsing for noisy/multi-format output
• Container Management: etcd now managed from NIXL container

9. Reporting & Metrics

• NCCL Comparison Reports: NCCL comparison reports with latency metrics
• Reusable Framework: Reusable comparison report framework for NIXL
• Multi-section CSV: Multi-section CSV format handling for AI Dynamo
• Configurable Reports: Configurable reports via scenario configuration
• DSE Trajectory Support: DSE trajectory support for single-sbatch mode

10. Development & Maintenance

• Environment Management: Added uv.lock for persistent environment management
• Code Cleanup: Removed deprecated NemoLauncher-based configurations
• Refactoring: NeMo recipes refactoring for better maintainability

11. Reliability & Error Handling

• Output Directory Handling: Improved output directory error handling (permissions, read-only filesystem)
• Graceful Error Handling: Graceful handling of missing tests with MissingTestError
• Environment Preservation: Better environment variable preservation (order maintained from system schema)
• Debug Logging: Enhanced debug logging for system config parsing errors

What's Changed (details)

• Support custom matgen args and set valid ppn by @amaslenn in https://github.com/NVIDIA/cloudai/pull/612
• Fix gres related directives for single sbatch mode by @amaslenn in https://github.com/NVIDIA/cloudai/pull/613
• Preserve the order of environment variables specified in the system schema by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/616
• Update docker_image_url separator from colon to hash by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/621
• Bump default version to v1.4 by @amaslenn in https://github.com/NVIDIA/cloudai/pull/622
• Update USER_GUIDE.md by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/623
• Update doc/ai_dynamo.md by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/624
• Update conf/common/test/nemo_run_llama3_8b.toml by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/625
• Replace PyTorch image tag from 24.02-py3 to 25.06-py3 in all conf TOML files by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/627
• Update doc/ai_dynamo.md by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/628
• Replace Nemo image tag from 24.12.rc3 to 25.04.rc2 in all conf TOML files by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/626
• Improve prepare_output_dir error handling for permissions and read-only fs by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/629
• Improve prepare_output_dir error handling for permissions and read-only fs (continued) by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/631
• Add GitRepo support to KubernetesInstaller with install/uninstall logic by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/634
• Use a shell script as the entry point for AI Dynamo by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/615
• Handle multi-section CSV format in AI Dynamo report generation by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/620
• Add multi-worker-per-node GPU slicing support with dynamic allocation by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/636
• Log mapping between AI Dynamo nodes and roles by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/617
• Updates for SlurmContainer workload by @amaslenn in https://github.com/NVIDIA/cloudai/pull/638
• Handle missing tests gracefully by adding MissingTestError to avoid backtrace by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/640
• Clean up src/cloudai/workloads/ai_dynamo/ai_dynamo.sh by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/639
• Preserve installables' state during apply_params_set() by @amaslenn in https://github.com/NVIDIA/cloudai/pull/643
• Control which env vars dumped for per-rand evaluation by @amaslenn in https://github.com/NVIDIA/cloudai/pull/642
• Align extra_env_vars definition in test and scenario by @amaslenn in https://github.com/NVIDIA/cloudai/pull/644
• Update USER_GUIDE.md by @TaekyungHeo in https://github.com/NVIDIA/cloudai/pull/646
• Add latency metric reporting for NCCL by @amaslenn in https://github.com/NVIDIA/cloudai/pull/645

*...