replicate/image-service

replicate/image-service

Description: Nydus - the Dragonfly image service, providing fast, secure and easy access to container images.

Language: Rust

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 1

Created: 2023-08-02T15:17:13Z

Pushed: 2023-08-08T12:39:37Z

Default branch: stable/v2.2-our-patches

Fork: yes

Parent repository: dragonflyoss/nydus

Archived: no

README:

Nydus: Dragonfly Container Image Service

  

Introduction

The nydus project implements a content-addressable filesystem on top of a RAFS format that improves the current OCI image specification, in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity.

The following benchmarking result shows the performance improvement compared with the OCI image for the container cold startup elapsed time on containerd. As the OCI image size increases, the container startup time of using Nydus image remains very short.


Nydus' key features include:

• Container images can be downloaded on demand in chunks for lazy pulling to boost container startup
• Chunk-based content-addressable data de-duplication to minimize storage, transmission and memory footprints
• Merged filesystem tree in order to remove all intermediate layers as an option
• in-kernel EROFS or FUSE filesystem together with overlayfs to provide full POSIX compatibility
• E2E image data integrity check. So security issues like "Supply Chain Attach" can be avoided and detected at runtime
• Compatible with the OCI artifacts spec and distribution spec, so nydus image can be stored in a regular container registry
• Native eStargz image support with remote snapshotter plugin nydus-snapshotter for containerd runtime.
• Various container image storage backends are supported. For example, Registry, NAS, Aliyun/OSS, S3.
• Integrated with CNCF incubating project Dragonfly to distribute container images in P2P fashion and mitigate the pressure on container registries
• Capable to prefetch data block before user IO hits the block thus to reduce read latency
• Record files access pattern during runtime gathering access trace/log, by which user abnormal behaviors are easily caught
• Access trace based prefetch table
• User I/O amplification to reduce the amount of small requests to storage backend.

Currently Nydus includes following tools:

| Tool | Description | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | nydusd | Nydus user-space daemon, it processes all fscache/FUSE messages from the kernel and parses Nydus images to fullfil those requests | | nydus-image | Convert a single layer of OCI format container image into a nydus format container image generating meta part file and data part file respectively | | nydusify | It pulls OCI image down and unpack it, invokes nydus-image create to convert image and then pushes the converted image back to registry and data storage | | nydusctl | Nydusd CLI client (nydus-image inspect), query daemon's working status/metrics and configure it | | ctr-remote | An enhanced containerd CLI tool enable nydus support with containerd ctr | | nydus-docker-graphdriver | [Experimental] Works as a docker remote graph driver to control how images and containers are stored and managed | | nydus-overlayfs | Containerd mount helper to invoke overlayfs mount with tweaking mount options a bit. So nydus prerequisites can be passed to vm-based runtime | | [nydus-backend-proxy](./contrib/nydus-backend-proxy/README.md) | A simple HTTP server to serve local directory as a blob backend for nydusd |

Currently Nydus is supporting the following platforms in container ecosystem:

| Type | Platform | Description | Status | | ------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | | Storage | Registry/OSS/S3/NAS | Support for OCI-compatible distribution implementations such as Docker Hub, Harbor, Github GHCR, Aliyun ACR, NAS, and Aliyun OSS-like object storage service | ✅ | | Storage/Build | Harbor | Provides a general service for Harbor to support acceleration image conversion based on kinds of accelerator like Nydus and eStargz etc | ✅ | | Distribution | Dragonfly | Improve the runtime performance of Nydus image even further with the Dragonfly P2P data distribution system | ✅ | | Build | Buildkit | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ | | Runtime | Kubernetes | Run Nydus image using CRI interface | ✅ | | Runtime | Containerd | Nydus Snapshotter, a containerd remote plugin to run…