NVIDIA/aistore v1.4.6

4.6

Repository: NVIDIA/aistore

Tag: v1.4.6

Published: 2026-05-28T00:20:25Z

Prerelease: no

Release notes: AIStore 4.6 is a short-cycle reliability and scalability release following 4.5.

The drivers are practical: stabilize get-batch under heavy ML-style stress, replace the list-objects startup grace period with an explicit two-phase commit (2PC) protocol, avoid cgroup-v2 broadcast throttling in large clusters, harden primary-restart bootstrap, guard against prefetch misconfiguration, and make prefetch/blob-downloader behavior easier to observe and control.

Get-Batch is the largest area of work. Under heavy ML-style load - many concurrent batch requests, large objects, slow or disconnecting clients - the prior implementation could leak work items, take too long to abort, or mishandle late arrivals after cleanup. The 4.6 changes restructure the work-item lifecycle around explicit ownership, receive-side pinning, progress-based cleanup timers, bounded abort latency, and stricter late-arrival rejection.

Remote list-objects, also referred to as R-flow, gets a structural fix. The temporary startup-grace workaround from 4.5 is replaced with an explicit begin/commit/abort sequence: targets that are about to receive pages prepare first, the randomly selected *designated target* starts paging only after the receive side is ready, and abort has a dedicated cleanup path.

Broadcast parallelism in containerized deployments is also improved. In deployments using cgroup-v2 CPU quotas, a proxy with a small visible CPU count could collapse one-to-many broadcasts to single-digit parallelism even when targeting dozens of targets. A dedicated cluster broadcast wait group now covers the mostly I/O-bound intra-cluster fanout paths: keepalive, rebalance, generic broadcasts, and xaction target-to-target control.

The intra-cluster control plane is faster on the hot path. Keepalives and broadcasts - invoked behind essentially every cluster operation - no longer register a timer, allocate a context, and defer a cancel on every call. A separate fix covers a former primary restarting without a local cluster map: rather than assuming the primary role from local configuration alone, the proxy probes alternative URLs and validates returned node-state-info first.

Prefetch and blob-downloader observability is another release driver. A single prefetch job can spawn many blob-download child jobs. The 4.6 release adds clearer parent/child reporting, per-xaction stats, bucket-keyed Prometheus metrics, and rejection counters so operators can understand progress and throttling behavior without correlating logs manually. Get-Batch and list-objects extend their control-message output in the same direction. The cross-cutting picture is summarized in [Observability](#observability).

No data-format migration is required. Operational and API behavior changes are summarized in [Upgrade Notes](#upgrade-notes).

---

Table of Contents

1. [Get-Batch](#get-batch) 2. [Remote List-Objects (R-flow)](#list-objects) 3. [Large-Cluster Broadcast Scalability](#large-cluster-broadcast-scalability) 4. [Intra-Cluster Control Plane](#intra-cluster-control-plane) 5. [Primary Restart (bootstrap)](#primary-restart-bootstrap) 6. [Prefetch and Blob Downloader](#prefetch-and-blob-downloader) 7. [Space Cleanup](#space-cleanup) 8. [Shard-Index Cleanup](#shard-index-cleanup) 9. [Xaction Go API](#xaction-go-api) 10. [ETL](#etl) 11. [Python SDK and aisloader](#python-sdk-and-aisloader) 12. [Other Fixes](#other-fixes) 13. [Documentation and Tools](#documentation-and-tools) 14. [Observability](#observability) 15. [Upgrade Notes](#upgrade-notes)

---

Get-Batch

Work-item lifecycle

Get-Batch is highly concurrent by design: a single x-moss xaction multiplexes many batch requests, each driven by a remote client. A slow or disconnected client can leave its work item parked on the target side - receivers still pinned, SGLs still held - while making no forward progress. We call these *abandoned* work items; detecting them reliably without falsely killing legitimately slow batches is one of the problems 4.6 addresses.

Work-item ownership is now an explicit three-state model rather than a single demand flag. Assembly claims a work item before consuming it; abandoned-work GC may clean only unowned work; and receive paths use pin/unpin discipline, so data and control receives cannot accidentally keep touching a work item that is already cleaning up or abandoned.

Abandoned work items are marked before cleanup, and new receives are rejected once cleanup begins. Cleanup waits for a stable receive/assembly quiet window, using last-receive and last-assembly progress timestamps rather than only the original start time. This matters for large batches that can legitimately run for a long time while still making forward progress.

The work-item memory pool (sync.Pool) is removed in favor of explicit receive-side pin/unpin. The housekeeper gets a dedicated x-moss maintenance callback, so abandoned-work cleanup no longer depends on the generic DemandBase idle/fini path, and each cleanup pass is capped so the housekeeper remains responsive.

The shared data-mover (SDM) receive-side lock ordering is tightened as well: the receiver-demux mutex is no longer held across Close(), and Rx drop now distinguishes SDM close from xaction abort.

Abort and late-arrival handling

Abort latency is now bounded. An unbounded wait-group.Wait() is replaced with a bounded poll, and drainRx is extracted with a brief mode for fast cleanup paths. Stuck receive-side work items are forced through cleanup via the abandoned path. Late assembly is rejected after x-moss is aborted or done; late Assemble calls after abort cleanup are tolerated; and internal stopped sentinels are treated as quiet assembly termination.

Stop/Abort terminal semantics are tightened. DemandBase.Stop is now idempotent and protected by its own atomic, and the common SetStopping returns its CAS result so get-batch can use it to eliminate a race against its own concurrent Abort. DemandBase.Stop is now called before waiting and quiescing.

*Get-from-neighbor* (GFN) is the internal target-to-target object-fetch path, used both during global rebalance, when objects…