anthropics/blobfile

anthropics/blobfile

Language: Python

License: Unlicense

Stars: 11

Forks: 7

Open issues: 0

Created: 2023-12-14T06:42:51Z

Pushed: 2024-02-02T00:07:02Z

Default branch: master

Fork: no

Archived: no

README:

blobfile

This is a library that provides a Python-like interface for reading local and remote files (only from blob storage), with an API similar to open() as well as some of the os.path and shutil functions. blobfile supports local paths, Google Cloud Storage paths (gs://), and Azure Blob Storage paths (az:/// or https://.blob.core.windows.net//).

The main function is BlobFile, which lets you open local and remote files that act more or less like local ones. There are also a few additional functions such as basename, dirname, and join, which mostly do the same thing as their os.path namesakes, only they also support GCS paths and ABS paths.

This library is inspired by TensorFlow's `gfile` but does not have exactly the same interface.

Installation

pip install blobfile

Usage

# write a file, then read it back

import blobfile as bf

with bf.BlobFile("gs://my-bucket-name/cats", "wb") as f:
f.write(b"meow!")

print("exists:", bf.exists("gs://my-bucket-name/cats"))

with bf.BlobFile("gs://my-bucket-name/cats", "rb") as f:
print("contents:", f.read())

There are also some [examples processing many blobs in parallel](docs/parallel_examples.md).

Here are the functions in blobfile:

• BlobFile - like open() but works with remote paths too, data can be streamed to/from the remote file. It accepts the following arguments:
• streaming:
• The default for streaming is True when mode is in "r", "rb" and False when mode is in "w", "wb", "a", "ab".
• streaming=True:
• Reading is done without downloading the entire remote file.
• Writing is done to the remote file directly, but only in chunks of a few MB in size. flush() will not cause an early write.
• Appending is not implemented.
• streaming=False:
• Reading is done by downloading the remote file to a local file during the constructor.
• Writing is done by uploading the file on close() or during destruction.
• Appending is done by downloading the file during construction and uploading on close().
• buffer_size: number of bytes to buffer, this can potentially make reading more efficient.
• cache_dir: a directory in which to cache files for reading, only valid if streaming=False and mode is in "r", "rb". You are reponsible for cleaning up the cache directory.
• file_size: size of the file being opened, can be specified directly to avoid checking the file size when opening the file. While this will avoid a network request, it also means that you may get an error when first reading a file that does not exist rather than when opening it. Only valid for modes "r" and "rb". This valid will be ignored for local files.

Some are inspired by existing os.path and shutil functions:

• copy - copy a file from one path to another, this will do a remote copy between two remote paths on the same blob storage service
• exists - returns True if the file or directory exists
• glob/scanglob - return files matching a glob-style pattern as a generator. Globs can have surprising performance characteristics when used with blob storage. Character ranges are not supported in patterns.
• isdir - returns True if the path is a directory
• listdir/scandir - list contents of a directory as a generator
• makedirs - ensure that a directory and all parent directories exist
• remove - remove a file
• rmdir - remove an empty directory
• rmtree - remove a directory tree
• stat - get the size and modification time of a file
• walk - walk a directory tree with a generator that yields (dirpath, dirnames, filenames) tuples
• basename - get the final component of a path
• dirname - get the path except for the final component
• join - join 2 or more paths together, inserting directory separators between each component

There are a few bonus functions:

• get_url - returns a url for a path (usable by an HTTP client without any authentication) along with the expiration for that url (or None)
• md5 - get the md5 hash for a path, for GCS this is often fast, but for other backends this may be slow. On Azure, if the md5 of a file is calculated and is missing from the file, the file will be updated with the calculated md5.
• set_mtime - set the modified timestamp for a file
• configure - set global configuration options for blobfile
• log_callback=default_log_fn: a log callback function log(msg: string) to use instead of printing to stdout. If you use parallel=True, you probably want to use a log callback function that is pickleable.
• connection_pool_max_size=32: the max size for each per-host connection pool
• max_connection_pool_count=10: the maximum count of per-host connection pools
• azure_write_chunk_size=8 * 2 ** 20: the size of blocks to write to Azure Storage blobs in bytes, can be set to a maximum of 100MB. This determines both the unit of request retries as well as the maximum file size, which is 50,000 * azure_write_chunk_size.
• google_write_chunk_size=8 * 2 ** 20: the size of blocks to write to Google Cloud Storage blobs in bytes, this only determines the unit of request retries.
• retry_log_threshold=0: set a retry count threshold above which to log failures to the log callback function
• retry_common_log_threshold=2: set a retry count threshold above which to log very common failures to the log callback function
• connect_timeout=10: the maximum amount of time (in seconds) to wait for a connection attempt to a server to succeed, set to None to wait forever
• read_timeout=30: the maximum amount of time (in seconds) to wait between consecutive read operations for a response from the server, set to None to wait forever
• output_az_paths=True: output az:// paths instead of using the https:// for azure
• use_azure_storage_account_key_fallback=False: fallback to storage account keys for azure containers, having this enabled requires listing your subscriptions and may run into 429 errors if you hit the low azure quotas for subscription listing
• get_http_pool=None: a function that returns a…