anthropics/s5cmd

anthropics/s5cmd

Language: Go

License: MIT

Stars: 37

Forks: 10

Open issues: 5

Created: 2023-02-27T17:08:04Z

Pushed: 2026-05-19T18:38:42Z

Default branch: master

Fork: no

Archived: no

README:

anthropic s5cmd fork

We have forked s5cmd in order to add GCS and WIF support so that we can continue to use this tool without the need for long-lived credentials, i.e. HMAC keys.

There are minor differences in the implementation from the original. See this PR for the motivating changes.

Development Guide

This project largely uses the standard go tooling.

Run tests with make test (basically just go test ./...).

Running token manager benchmarks

To benchmark the token manager performance:

go test -v ./storage -run='^$' -bench=Token

This runs the various token manager benchmarks to measure performance in different scenarios including token acquisition, parallel access, and context cancellation handling.

Example of benchmarking with hyperfine

hyperfine --export-markdown bench.md -u second --runs 10 --warmup 0 -n s5cmd-old-ff92bdc -n s5cmd-new-HEAD --show-output '~/code/benchmark-s5cmd/old/s5cmd cp testfile-s5cmd-10mb s3://serum-intake-waiver-espalier/s5cmd-benchmarks/' '~/code/benchmark-s5cmd/new/s5cmd cp testfile-s5cmd-10mb s3://serum-intake-waiver-espalier/s5cmd-benchmarks/'

s5cmd's preexisting benchmarking script, non-functional

(even though rr tried to fix it)

Benchmark two versions of s5cmd with:

# bucket in seceng playground account, us-east-1
BUCKET_NO_S3_PREFIX="serum-intake-waiver-espalier" \
OLD="HEAD~1" \
NEW="HEAD" \
python ./benchmark/bench.py -b $BUCKET_NO_S3_PREFIX -s $OLD $NEW -hf ' --show-output'

OLD README FOLLOWS

Overview

s5cmd is a very fast S3 and local filesystem execution tool. It comes with support for a multitude of operations including tab completion and wildcard support for files, which can be very handy for your object storage workflow while working with large number of files.

There are already other utilities to work with S3 and similar object storage services, thus it is natural to wonder what s5cmd has to offer that others don't.

In short, *s5cmd offers a very fast speed.* Thanks to Joshua Robinson for his study and experimentation on s5cmd; to quote his medium post: > For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. >For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd >and aws-cli can only reach 85 MB/s and 375 MB/s respectively.

If you would like to know more about performance of s5cmd and the reasons for its fast speed, refer to [benchmarks](./README.md#Benchmarks) section

Features


s5cmd supports wide range of object management tasks both for cloud storage services and local filesystems.

• List buckets and objects
• Upload, download or delete objects
• Move, copy or rename objects
• Set Server Side Encryption using AWS Key Management Service (KMS)
• Set Access Control List (ACL) for objects/files on the upload, copy, move.
• Print object contents to stdout
• Select JSON records from objects using SQL expressions
• Create or remove buckets
• Summarize objects sizes, grouping by storage class
• Wildcard support for all operations
• Multiple arguments support for delete operation
• Command file support to run commands in batches at very high execution speeds
• Dry run support
• S3 Transfer Acceleration support
• Google Cloud Storage (and any other S3 API compatible service) support
• Structured logging for querying command outputs
• Shell auto-completion
• S3 ListObjects API backward compatibility

Installation

Official Releases

Binaries

The Releases page provides pre-built binaries for Linux, macOS and Windows.

Homebrew

For macOS, a homebrew tap is provided:

brew install peak/tap/s5cmd

Unofficial Releases (by Community)

 > Warning > These releases are maintained by the community. They might be out of date compared to the official releases.

MacPorts

You can also install s5cmd from MacPorts on macOS:

sudo port selfupdate sudo port install s5cmd

Conda

s5cmd is included in the [conda-forge]( https://conda-forge.org ) channel, and it can be downloaded through the Conda.

> Installing s5cmd from the conda-forge channel can be achieved by adding conda-forge to your channels with: > `` > conda config --add channels conda-forge > conda config --set channel_priority strict > > > Once the conda-forge channel has been enabled, s5cmd can be installed with conda: > > > conda install s5cmd > `` ps. Quoted from s5cmd feedstock. You can also find further instructions on its README.

FreeBSD

On FreeBSD you can install s5cmd as a package:

pkg install s5cmd

or via ports:

cd /usr/ports/net/s5cmd
make install clean

Build from source

You can build s5cmd from source if you have Go 1.19+ installed.

go install github.com/peak/s5cmd/v2@master

⚠️ Please note that building from master is not guaranteed to be stable since development happens on master branch.

Docker

Hub

$ docker pull peakcom/s5cmd $ docker run --rm -v ~/.aws:/root/.aws peakcom/s5cmd

ℹ️ /aws directory is the working directory of the image. Mounting your current working directory to it allows you to run s5cmd as if it was installed in your system;

docker run --rm -v $(pwd):/aws -v ~/.aws:/root/.aws peakcom/s5cmd

Build

$ git clone https://github.com/peak/s5cmd && cd s5cmd $ docker build -t s5cmd . $ docker run --rm -v ~/.aws:/root/.aws s5cmd

Usage

s5cmd supports multiple-level wildcards for all S3 operations. This is achieved by listing all S3 objects with the prefix up to the first wildcard, then filtering the results in-memory. For example,…