NVIDIA/aistore v1.4.4

4.4

Repository: NVIDIA/aistore

Tag: v1.4.4

Published: 2026-04-08T17:14:24Z

Prerelease: no

Release notes: AIStore 4.4 is a short release cycle focused on runtime correctness, container awareness, and consolidation.

This release is the first to deliver full cgroup v2 support, improving AIS behavior in constrained and containerized deployments. CPU and memory accounting are now cgroup-aware, container detection and startup initialization are more robust, and CPU utilization is reported as a smoothed moving average rather than an instantaneous sample.

4.4 also promotes native bucket inventory (NBI) into the default AIS path for S3-compatible inventory workflows. The release adds non-recursive inventory listing, removes the legacy S3-specific inventory implementation, and updates the S3 compatibility layer to route inventory-backed requests through NBI. NBI should now be regarded as stable.

Additional changes improve S3 bucket region discovery and error reporting, add per-bucket OCI regions and instance principal authentication, introduce per-bucket user-defined custom metadata, and update the CLI, documentation, tracing, and build tooling.

This release includes roughly 50 commits since 4.3 and is fully backward compatible.

---

Table of Contents

1. [Runtime: cgroup v2, CPU, and Memory](#runtime-cgroup-v2-cpu-and-memory) 2. [Native Bucket Inventory (NBI)](#native-bucket-inventory-nbi) 3. [S3 Compatibility and Remote Bucket Discovery](#s3-compatibility-and-remote-bucket-discovery) 4. [OCI Backend](#oci-backend) 5. [Bucket Metadata and BMD Changes](#bucket-metadata-and-bmd-changes) 6. [CLI](#cli) 7. [Python SDK and ETL](#python-sdk-and-etl) 8. [Tracing](#tracing) 9. [Website](#website) 10. [Documentation](#documentation) 11. [Build, CI, and Tools](#build-ci-and-tools) 12. [Upgrade Notes](#upgrade-notes)

---

Runtime: cgroup v2, CPU, and Memory

The main feature of 4.4 is full cgroup v2 support, with a broader runtime refactor to make AIS behave correctly in standard containers and restricted Kubernetes-style environments.

See Also

• AIS in Containerized Environments

The work starts with startup and environment detection. AIS now initializes CPU and memory accounting earlier and more explicitly via sys.Init(), revises container auto-detection with additional heuristics, and adds the ForceContainerCPUMem feature flag to override failed detection when needed. The initialization path also applies container-aware CPU counting and GOMAXPROCS, and avoids retrying failed cgroup parsing later at runtime.

On the memory side, AIS now supports cgroup-v2 memory reporting based on memory.max, memory.current, and memory.stat, while keeping memory.stat best-effort so that missing or unreadable auxiliary details do not turn into fatal runtime failures. Host and container memory readers have been refactored into stateless helpers, and related error reporting has been cleaned up and prefixed consistently.

On the CPU side, bare-metal /proc/stat parsing is corrected to use specific enumerated fields and avoid double-counting. The runtime now maintains global cached CPU state and reports CPU utilization as a smoothed, time-aware moving average rather than an instantaneous sample. The new model is better suited for operator-facing reporting and long-running process decisions, and it also supports CPU throttling (cgroup v2).

The CLI reflects these changes directly. ais show cluster now shows SYS CPU(%) instead of load average by default, moves load averages behind --verbose, and adds THROTTLED(%) when the displayed proxy or target section contains at least one non-zero cgroup-v2 throttling value.

The last part of the work addresses deployment realism. On Kubernetes, cgroup-v2 files may live under per-pod subtrees rather than fixed global paths, so AIS now resolves /sys/fs/cgroup paths at init time instead of hardcoding them. Startup logging also returns a more descriptive container tag, and the implementation includes explicit constrained-container testing and documentation.

Commits

• dd03cae2c: sys: bare-metal CPU usage parsing (fix)
• 1b1397ee3: sys: cpu and memory (part two; major)
• 3fb3aeb7e: sys: cpu and memory (support cgroup-v2 memory stats)
• ac6c9461a: sys: cpu and memory (cgroup v2 in a constrained container)
• 074554a72: sys: CPU utilization is now a moving average; [API change]
• 30e7f430a: sys: cpu and memory (resolve /sys/fs/cgroup/ paths at init time)
• 366a8906e: cli: show CPU utilization (EMA); load-averages only when verbose

---

Native Bucket Inventory (NBI)

[Native bucket inventory](/docs/nbi.md) was introduced in 4.3. In 4.4, it becomes the clear and supported path for AIS-managed inventory-backed listing.

The most visible functional addition is support for non-recursive inventory listing via --nr. That closes an important gap for users who want inventory-backed listing while preserving delimiter-style behavior rather than flattening the namespace unconditionally.

Just as importantly, 4.4 removes the older S3-specific inventory implementation and routes S3-compatible inventory-backed requests to NBI instead. This includes removal of deprecated S3-only backend hooks, old R-flow plumbing, and deprecated CLI/API pieces such as --s3-inventory. At the S3 compatibility layer, AIS keeps the existing Ais-Bucket-Inventory and Ais-Inv-Name headers, but now interprets them through NBI with validation.

With 4.4, NBI is no longer experimental. The docs are updated accordingly; 4.4 adds benchmark tooling and documentation around large-bucket listing to support that shift in status.

See Also

• Native Bucket Inventory: Up to **17x** Faster Remote Bucket Listing

Commits

• ddbca5e78: native bucket inventory: support non-recursive mode (--nr)

*…