Lightning-AI/litData v0.2.55

LitData v0.2.55

Repository: Lightning-AI/litData

Tag: v0.2.55

Published: 2025-09-19T15:47:59Z

Prerelease: no

Release notes: Lightning AI ⚡ is excited to announce the release of LitData v0.2.55

Highlights

[Fixed] Writing compressed data to a lighting_storage folder

This release focuses on fixing errors when writing compressed output data to a lightning_storage folder. Previously, a code snippet like the following would break.

from litdata import StreamingDataset, StreamingDataLoader, optimize
import time

def should_keep(data):
if data % 2 == 0:
yield data

if __name__ == "__main__":
output_dir = "/teamspace/lightning_storage/my-folder-1/output"
optimize(
fn=should_keep,
inputs=list(range(500)),
output_dir=output_dir,
chunk_bytes="64MB",
num_workers=4,
compression="zstd", # Previously, this would cause an error
)
time.sleep(20)
dataset = StreamingDataset(output_dir)
dataloader = StreamingDataLoader(dataset, batch_size=32, num_workers=4)
for _ in dataloader:
# process code here
pass

Changes

Fixed

• Fix errors when using compression and r2 in optimize() by @pwgardipee in https://github.com/Lightning-AI/litData/pull/715

Changed

• Remove s5cmd from the R2 downloader by @pwgardipee in https://github.com/Lightning-AI/litData/pull/714

Chores

• chore(ci): Add step to minimize uv cache in CI workflow by @bhimrazy in https://github.com/Lightning-AI/litData/pull/713

Full Changelog: https://github.com/Lightning-AI/litData/compare/v0.2.54...v0.2.55

🧑‍💻 Contributors

We thank all folks who submitted issues, features, fixes and doc changes. It's the only way we can collectively make LitData better for everyone, nice job!

Key Contributors

@pwgardipee @bhimrazy

Thank you ❤️ and we hope you'll keep them coming!