NVIDIA/aistore v1.4.1

4.1

Repository: NVIDIA/aistore

Tag: v1.4.1

Published: 2025-12-05T23:10:01Z

Prerelease: no

Release notes: AIStore 4.1 delivers upgrades across retrieval, security, and cluster operation. The GetBatch API is significantly expanded for ML training workloads, with client-side streaming, improved resilience and error handling under resource shortages. Authentication is redesigned with OIDC support, structured JWT validation, and cluster-key HMAC signing for HTTP redirects. This release also adds the rechunk job for converting datasets between monolithic and chunked layouts, unifies multipart-upload behavior across all cloud backends, and enhances the blob downloader with load-aware throttling. The Python SDK advances to v1.18 with a redesigned Batch API and revised timeout configuration. Configuration validation is strengthened throughout, including automatic migration from v4.0 auth settings.

This release arrives with over 200 commits since v4.0 and maintains backward compatibility, supporting rolling upgrades.

Table of Contents

1. [GetBatch: Distributed Multi-Object Retrieval](#getbatch-distributed-multi-object-retrieval) 2. [Authentication and Security](#authentication-and-security) 3. [Chunked Objects](#chunked-objects) 4. [Blob Downloader](#blob-downloader) 5. [Rechunk Job](#rechunk-job) 6. [Unified Load and Throttling](#unified-load-and-throttling) 7. [Transport Layer](#transport-layer) 8. [Multipart Upload](#multipart-upload) 9. [Python SDK](#python-sdk) 10. [S3 Compatibility](#s3-compatibility) 11. [Build System and Tooling](#build-system-and-tooling) 12. [Xaction Lifecycle](#xaction-lifecycle) 13. [ETL and Transform Pipeline](#etl-and-transform-pipeline) 14. [Observability](#observability) 15. [Configuration Changes](#configuration-changes) 16. [Tools: aisloader](#tools-aisloader)

---

GetBatch: Distributed Multi-Object Retrieval

The GetBatch workflow now has a robust implementation across the cluster. Retrieval is streaming-oriented, supports multi-bucket batches, and includes tunable soft-error handling. The request path incorporates load-based throttling and may return HTTP 429 ("too many requests") when the system is under severe pressure. Memory and disk pressure are taken into account, and connection resets are handled transparently.

Configuration is available via a new "get_batch" section:

{
"max_wait": "30s", // Wait time for remote targets (range: 1s-1m)
"warmup_workers": 2, // Pagecache read-ahead workers (-1=disabled, 0-10)
"max_soft_errs": 6 // Recoverable error limit per request
}

Observability has improved through consolidated counters, Prometheus metrics, and clearer status reporting. Client and tooling updates include a new Batch API in the Python SDK, extended aisloader support, and ansible composer playbooks for distributed benchmarks.

> Reference: https://github.com/NVIDIA/aistore/blob/main/docs/get_batch.md

Commit Highlights

• 5e3382dfa: Refine throttling under memory/disk pressure
• d093b5c0f: Add Prometheus metrics for GetBatch
• e71ad3e3c: Consolidate statistics and add counters
• 2ab4c289a: Handle stream-breakages (ErrSBR); per-request polling and cleanup
• 1c8dc4b29: Recover from connection drops/resets via SharedDM
• 9316804d5: Add client-side streaming GetBatch API (breaking change)
• 174931299: Avoid aborting x-moss; rename internal intra-cluster headers

---

Authentication and Security

AIS v4.1 introduces a standardized JWT validation model, reorganized configuration for external authentication, and an expanded cluster-key mechanism for securing intra-cluster redirects. Together, these changes provide clearer semantics, better interoperability with third-party identity providers, and more uniform behavior across proxies and targets.

JWT Validation Model and Token Requirements

AIS uses JWTs to both authenticate and authorize requests. Version 4.1 formalizes this process and documents the complete validation flow in `auth_validation.md`.

Tokens may be issued by the first-party AuthN service or by compatible third-party identity providers (Keycloak, Auth0, custom OAuth services). AIS makes authorization decisions directly from JWT claims; no external role lookups are performed during request execution.

When auth.enabled=true is set in the cluster configuration, proxies validate tokens before routing requests to targets; targets verify redirect signatures when cluster-key signing is enabled.

Token Requirements

AIS accepts tokens signed with supported HMAC or RSA algorithms (HS256/384/512, RS256/384/512). All tokens must include the standard sub and exp claims; aud and iss are validated when required by configuration.

AIS also recognizes several AIS-specific claims:

• admin: Full administrative access; overrides all other claims
• clusters: Cluster-scoped permissions (specific cluster UUID or wildcard)
• buckets: Bucket-scoped permissions tied to individual buckets within a cluster

Cluster and bucket permissions use the access-flag bitmask defined in api/apc/access.go.

Signature Verification: Static or OIDC

AIS supports two mutually exclusive approaches for signature verification:

1. Static verification (`auth.signature`)

• HMAC (shared secret) or RSA public-key–based
• Suitable for the AIS AuthN service or controlled token issuers
• Verifies tokens using the configured secret or public key

2. OIDC verification (`auth.oidc`)

• Automatic discovery using /.well-known/openid-configuration
• JWKS retrieval and caching with periodic refresh
• Validates issuer (iss) against allowed_iss
• Supports custom CA bundles for TLS verification

Both modes accept standard Authorization: Bearer headers and AWS-compatible X-Amz-Security-Token.

Authentication Flow

1. Extract token from Authorization or X-Amz-Security-Token. 2. Validate signature via static credentials or OIDC...