coreweave/sriov-network-device-plugin

coreweave/sriov-network-device-plugin

Description: SRIOV network device plugin for Kubernetes

Language: Go

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 4

Created: 2022-06-29T15:47:06Z

Pushed: 2025-02-16T19:45:26Z

Default branch: coreweave

Fork: yes

Parent repository: k8snetworkplumbingwg/sriov-network-device-plugin

Archived: no

README:

SR-IOV Network Device Plugin for Kubernetes


Table of Contents

• [SR-IOV Network Device Plugin](#sr-iov-network-device-plugin)
• [Features](#features)
• [Supported SR-IOV NICs](#supported-sr-iov-nics)
• [Quick Start](#quick-start)
• [Install SR-IOV CNI](#install-sr-iov-cni)
• [Get SR-IOV Network Device Plugin container image](#get-sr-iov-network-device-plugin-container-image)
• [Install SR-IOV Network Device Plugin](#install-sr-iov-network-device-plugin)
• [Install one compatible CNI meta plugin](#install-one-compatible-cni-meta-plugin)
• [Configurations](#configurations)
• [Config parameters](#config-parameters)
• [Command line arguments](#command-line-arguments)
• [Assumptions](#assumptions)
• [Workflow](#workflow)
• [Example deployments](#example-deployments)
• [Deploy the Device Plugin](#deploy-the-device-plugin)
• [Deploy SR-IOV workloads when Multus is used](#deploy-sr-iov-workloads-when-multus-is-used)
• [Deploy SR-IOV workloads when DANM is used](#deploy-sr-iov-workloads-when-danm-is-used)
• [Pod device information](#pod-device-information)
• [Virtual Deployments Support](#virtual-deployments-support)
• [Multi Architecture Support](#multi-architecture-support)
• [Issues and Contributing](#issues-and-contributing)

SR-IOV Network Device Plugin

The SR-IOV Network Device Plugin is Kubernetes device plugin for discovering and advertising networking resources in the form of SR-IOV virtual functions (VFs) and PCI physical functions (PFs) available on a Kubernetes host.

Features

• Handles SR-IOV capable/not-capable devices (NICs and Accelerators alike)
• Supports devices with both Kernel and userspace (UIO and VFIO) drivers
• Allows resource grouping using "Selector"
• User configurable resourceName
• Detects Kubelet restarts and auto-re-register
• Detects Link status (for Linux network devices) and updates associated VFs health accordingly
• Extensible to support new device types with minimal effort if not already supported
• Works within virtual deployments of Kubernetes that do not have virtualized-iommu support (VFIO No-IOMMU support)

To deploy workloads with SR-IOV VF or PCI PF, this plugin needs to work together with the following two CNI components:

• Any CNI meta plugin supporting Device Plugin based network provisioning (Multus CNI, or DANM)

• Retrieves allocated network device information of a Pod

• A CNI capable of consuming the network device allocated to the Pod
• SR-IOV CNI (for SR-IOV VFs)
• During Pod creation, plumbs allocated SR-IOV VF to a Pods network namespace using VF information given by the meta plugin
• On Pod deletion, reset and release the VF from the Pod
• Host device CNI (for PCI PFs)
• During Pod creation, plumbs the allocated network device to the Pods network namespace using device information given by the meta plugin
• On Pod deletion, reset and release the allocated network device from the Pod

Please follow the [Quick Start](#quick-start) for multi network interface support in Kubernetes.

Supported SR-IOV NICs

The following NICs were tested with this implementation. However, other SR-IOV capable NICs should work as well.

• Intel® E800 Series
• Intel® X700 Series
• Intel® 82599ES
• Mellanox ConnectX-4®
• Mellanox Connectx-4® Lx EN Adapter
• Mellanox ConnectX-5®
• Mellanox ConnectX-5® Ex
• Mellanox ConnectX-6®
• Mellanox ConnectX-6® Dx

Quick Start

Creating SR-IOV Virtual Functions

Before starting the SR-IOV Network Device Plugin you will need to create SR-IOV Virtual Functions on your system. [The VF Setup doc will guide you through that process.](docs/vf-setup.md)

Install SR-IOV CNI

See the SR-IOV CNI repository for build and installation instructions. Supported from SR-IOV CNI release 2.0+.

Get SR-IOV Network Device Plugin container image

GitHub

$ docker pull ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin:latest

Build image locally

$ make image

> On a successful build, a docker image with tag ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin:latest will be created. You will need to build this image on each node. Alternatively, you could use a local docker registry to host this image.

Install SR-IOV Network Device Plugin

Deploy config map

Create a ConfigMap that defines SR-IOV resource pool configuration

> Make sure to update the 'config.json' entry in the configMap data to reflect your resource configuration for the device plugin. See [Configurations](#configurations) section for supported configuration parameters.

$ kubectl create -f deployments/configMap.yaml

Deploy daemonset

$ kubectl create -f deployments/k8s-v1.16/sriovdp-daemonset.yaml

> For K8s version v1.15 or earlier use deployments/k8s-v1.10-v1.15/sriovdp-daemonset.yaml instead.

Install one compatible CNI meta plugin

A compatible CNI meta-plugin installation is required for SR-IOV CNI plugin to be able to get allocated VF's deviceID in order to configure it.

Option 1 - Multus

##### Install Multus Please refer to Multus Quickstart Installation Guide to install Multus.

##### Network Object CRDs

Multus uses Custom Resource Definitions(CRDs) for defining additional network attachements. These network attachment CRDs follow the standards defined by K8s Network Plumbing Working Group(NPWG). Please refer to Multus documentation for more information. 1. Create the SR-IOV Network CRD

$ kubectl create -f deployments/sriov-crd.yaml

Option 2 - DANM

This section explains an example deployment of SR-IOV Network Device Plugin in Kubernetes if you choose DANM as your meta plugin.

##### Install DANM Refer to [DANM deployment…