NVIDIA/aistore v1.4.0

4.0

Repository: NVIDIA/aistore

Tag: v1.4.0

Published: 2025-10-07T20:20:25Z

Prerelease: no

Release notes: AIStore 4.0 is a major release that introduces a v2 object-metadata format and a chunked object representation (objects stored and managed as multiple chunks). This release also adds native multipart upload and a new [GetBatch API](#getbatch-api-ml-endpoint) for high-throughput retrieval of batches (objects and/or archived files).

In-cluster ETL has been extended with _ETL pipelines_, allowing users to chain transformations without intermediate buckets. Observability in 4.0 consolidates on Prometheus (as the sole monitoring backend) and adds disk-level capacity alerts

All subsystems, extensions, and modules were updated to support the new functionality. The CLI adds a cluster dashboard, improved feature-flag management, and numerous usability improvements. Configuration updates include a new chunks section and additional knobs to support clusters with hundreds of millions of objects and to improve runtime throttling.

This release arrives with nearly 300 commits since the previous 3.31 and maintains compatibility with the previous version, supporting rolling upgrades.

Table of Contents

• [Object Metadata Version 2](#object-metadata-version-2)
• [Chunked Objects](#chunked-objects)
• [Native API: Multipart Upload (unified implementation)](#native-api-multipart-upload-unified-implementation)
• [GetBatch API (ML Endpoint)](#getbatch-api-ml-endpoint)
• [ETL](#etl)
• [Observability and Metrics](#observability-and-metrics)
• [Python SDK 1.16](#python-sdk-116)
• [CLI](#cli)
• [Configuration Changes](#configuration-changes)
• [Performance Optimizations](#performance-optimizations)
• [Build, CI/CD, and Testing](#build-cicd-and-testing)
• [Documentation](#documentation)

---

Object Metadata Version 2

For the first time since AIStore's inception in 2018, the on-disk object metadata format has been upgraded.

Metadata v2 is now the default for all new writes, while v1 remains fully supported for backward compatibility. The system reads both versions seamlessly.

What's New

The upgrade introduces persistent bucket identity and durable flags. Each object now stores its bucket ID (BID) alongside metadata. On load, AIStore checks the stored BID against the current bucket's BID from BMD (bucket metadata). A mismatch marks the object as defunct and evicts it from the cache, enforcing strong referential integrity between every object and the exact bucket generation it was written to.

The previous format had limited room for new features. Version 2 adds a dedicated 8-byte field for future capabilities (e.g., storage-class, compression, encryption, write-back). The system now returns explicit, typed errors (for example, ErrLmetaCorrupted) when metadata is damaged or unparseable, improving troubleshooting and debuggability.

Legacy flags (filename-too-long, lom-is-chunked) keep their original on-disk placement for backward compatibility, but we no longer carve bits out of the BID for new features.

The integrity model is now stronger: BID and flags are verified on load (before any object access). If the bucket is missing or has a different BID, loads fail with explicit errors (ErrBckNotFound, ErrObjDefunct). From the user's perspective, a defunct object effectively disappears, though it remains on disk until the next space-cleanup.

The serialized layout now packs BID and flags alongside checksum and other fields. The (future-proof) format remains compact.

No manual migration is required. Clusters continue to read v1 metadata while all new writes automatically persist v2.

Commit Highlights

• ae9ed1d2: Introduce LOM (Local Object Metadata) v2 with BID persistence and durable flags
• cdd3beef: Enforce defunct object detection when BID mismatches current bucket
• b543dab7: Extend serialized layout with new packed fields (LID, flags)
• 12ac88fe: Update error handling and unpack() logic for v2 metadata

Another closely related and notable on-disk metadata change is chunked objects and chunk manifests - see [Chunked Objects](#chunked-objects).

---

Chunked Objects

Version 4.0 introduces chunked objects as a new persistent storage format. The previous limitation that required storing all objects (small or large) as single monoliths has been removed. For chunked objects, AIStore maintains a _chunk manifest_ that describes their content: chunk sizes, checksums, and ordering. The manifest itself is compressed and protected by a checksum.

Each in-progress [multipart upload](#native-api-multipart-upload-unified-implementation) creates a uniquely identified partial manifest, tied to the bucket, object name, and upload ID.

Multiple uploads can proceed in parallel, with each upload identified by its own manifest ID. Partial manifests are persistent, and the cluster automatically checkpoints them after every N chunks (configured via the "checkpoint_every" setting below):

{
"chunks": {
"objsize_limit": 0,
"chunk_size": "1GiB",
"checkpoint_every": 4
}
}

> The full set of chunk-related options is described in the [Configuration Changes](#configuration-changes) section.

At any given time, however, only one completed manifest exists for an object, serving as the current object version.

There is no limit — hard or soft — on the number of chunks. Chunks are sequentially numbered and distributed across mountpaths, while the manifest includes fields such as MD5 and ETag for S3 compatibility and reserved flags for future extensions like compression or encryption.

Interrupted or abandoned uploads are also cleaned up automatically. Any orphaned chunks or manifests are discovered and removed by the space-cleanup job, which runs on its own when a mountpath goes out of space, but can just as easily be invoked by an admin at any time:

$ ais space-cleanup --help

For users and applications, chunked objects are fully transparent. Chunked and monolithic formats coexist side by side, and the same is true across both the S3-compatible and native APIs. Standard GETs, range reads, and archive reads all continue to work out of the box.

> Note: Throughout this document,...