coreweave/docker-registry-proxy

coreweave/docker-registry-proxy

Description: An HTTPS Proxy for Docker providing centralized configuration and caching of any registry (quay.io, DockerHub, k8s.gcr.io)

Language: Shell

License: Apache-2.0

Stars: 16

Forks: 10

Open issues: 6

Created: 2021-05-13T17:51:07Z

Pushed: 2025-08-27T14:39:31Z

Default branch: coreweave

Fork: yes

Parent repository: rpardini/docker-registry-proxy

Archived: no

README:

TL,DR

A caching proxy for Docker; allows centralised management of (multiple) registries and their authentication; caches images from *any* registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting.

0.6.5: Updated late February 2025 for the "2nd Docker Apocalypse"

Docker, Inc has announced a 2nd apocalypse for 1st of March'25 (it has already been pushed back to April). This has caused a new surge of interest in this project; in response I've updated all dependencies to the latest versions, added a Test matrix, merged some pull requests (including DISABLE_IPV6=true, which was a long-standing request), and updated the documentation.

Many thanks to all the contributors over the years; I've no intention of abandoning this project -- please keep sending and updating your PRs.

NEW: avoiding DockerHub Pull Rate Limits with Caching

Starting November 2nd, 2020, DockerHub will supposedly start rate-limiting pulls, also known as the _Docker Apocalypse_. The main symptom is Error response from daemon: toomanyrequests: Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/ during pulls. Many unknowing Kubernetes clusters will hit the limit, and struggle to configure imagePullSecrets and imagePullPolicy.

This proxy can be configured with the env var ENABLE_MANIFEST_CACHE=true which provides configurable caching of the manifest requests that DockerHub throttles. You can then fine-tune other parameters to your needs. Together with the possibility to centrally inject authentication (since 0.3x), this is probably one of the best ways to bring relief to your distressed cluster, while at the same time saving lots of bandwidth and time.

Note: enabling manifest caching, in its default config, effectively makes some tags immutable. Use with care. The configuration ENVs are explained in the [Dockerfile](./Dockerfile), relevant parts included below.

# Manifest caching tiers. Disabled by default, to mimick 0.4/0.5 behaviour.
# Setting it to true enables the processing of the ENVs below.
# Once enabled, it is valid for all registries, not only DockerHub.
# The envs *_REGEX represent a regex fragment, check entrypoint.sh to understand how they're used (nginx ~ location, PCRE syntax).
ENV ENABLE_MANIFEST_CACHE="false"

# 'Primary' tier defaults to 10m cache for frequently used/abused tags.
# - People publishing to production via :latest (argh) will want to include that in the regex
# - Heavy pullers who are being ratelimited but don't mind getting outdated manifests should (also) increase the cache time here
ENV MANIFEST_CACHE_PRIMARY_REGEX="(stable|nightly|production|test)"
ENV MANIFEST_CACHE_PRIMARY_TIME="10m"

# 'Secondary' tier defaults any tag that has 3 digits or dots, in the hopes of matching most explicitly-versioned tags.
# It caches for 60d, which is also the cache time for the large binary blobs to which the manifests refer.
# That makes them effectively immutable. Make sure you're not affected; tighten this regex or widen the primary tier.
ENV MANIFEST_CACHE_SECONDARY_REGEX="(.*)(\d|\.)+(.*)(\d|\.)+(.*)(\d|\.)+"
ENV MANIFEST_CACHE_SECONDARY_TIME="60d"

# The default cache duration for manifests that don't match either the primary or secondary tiers above.
# In the default config, :latest and other frequently-used tags will get this value.
ENV MANIFEST_CACHE_DEFAULT_TIME="1h"

What?

Essentially, it's a man in the middle: an intercepting proxy based on nginx, to which all docker traffic is directed using the HTTPS_PROXY mechanism and injected CA root certificates.

The main feature is Docker layer/image caching, including layers served from S3, Google Storage, etc.

As a bonus it allows for centralized management of Docker registry credentials, which can in itself be the main feature, eg in Kubernetes environments.

You configure the Docker clients (_err... Kubernetes Nodes?_) once, and then all configuration is done on the proxy -- for this to work it requires inserting a root CA certificate into system trusted root certs.

master/:latest is unstable/beta

• :latest and :latest-debug Docker tag is unstable, built from master, and amd64-only
• Production/stable is 0.6.5, see 0.6.5 tag on Github - this image is multi-arch amd64/arm64

Also hosted on GitHub Container Registry (ghcr.io)

• DockerHub image is at rpardini/docker-registry-proxy:
• GitHub image is at ghcr.io/rpardini/docker-registry-proxy:
• Since 0.5.x, they both carry the same images
• This can be useful if you're already hitting DockerHub's rate limits and can't pull the proxy from DockerHub

Usage (running the Proxy server)

• Run the proxy on a host close (network-wise: high bandwidth, same-VPC, etc) to the Docker clients
• Expose port 3128 to the network
• Map volume /docker_mirror_cache for up to CACHE_MAX_SIZE (32gb by default) of cached images across all cached registries
• Map volume /ca, the proxy will store the CA certificate here across restarts. Important this is security sensitive.
• Env ALLOW_PUSH : This bypasses the proxy when pushing, default to false - if kept to false, pushing will not work. For more info see this commit.
• Env CACHE_MAX_SIZE (default 32g): set the max…