togethercomputer/sriov-network-operator

togethercomputer/sriov-network-operator

Description: Operator for provisioning and configuring SR-IOV CNI plugin and device plugin

Language: Go

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 7

Created: 2024-09-18T19:21:58Z

Pushed: 2026-04-10T21:13:07Z

Default branch: master

Fork: yes

Parent repository: k8snetworkplumbingwg/sriov-network-operator

Archived: no

README:

sriov-network-operator

The Sriov Network Operator is designed to help the user to provision and configure SR-IOV CNI plugin and Device plugin in the Openshift cluster.

Motivation

SR-IOV network is an optional feature of an Openshift cluster. To make it work, it requires different components to be provisioned and configured accordingly. It makes sense to have one operator to coordinate those relevant components in one place, instead of having them managed by different operators. And also, to hide the complexity, we should provide an elegant user interface to simplify the process of enabling SR-IOV.

Features

• Initialize the supported SR-IOV NIC types on selected nodes.
• Provision/upgrade SR-IOV device plugin executable on selected node.
• Provision/upgrade SR-IOV CNI plugin executable on selected nodes.
• Manage configuration of SR-IOV device plugin on host.
• Generate net-att-def CRs for SR-IOV CNI plugin
• Supports operation in a virtualized Kubernetes deployment
• Discovers VFs attached to the Virtual Machine (VM)
• Does not require attached of associated PFs
• VFs can be associated to SriovNetworks by selecting the appropriate PciAddress as the RootDevice in the SriovNetworkNodePolicy

Quick Start

For more detail on installing this operator, refer to the [quick-start](doc/quickstart.md) guide.

API

The SR-IOV network operator introduces following new CRDs:

• SriovNetwork

• OVSNetwork

• SriovNetworkNodeState

• SriovNetworkNodePolicy

SriovNetwork

A custom resource of SriovNetwork could represent the a layer-2 broadcast domain where some SR-IOV devices are attach to. It is primarily used to generate a NetworkAttachmentDefinition CR with an SR-IOV CNI plugin configuration.

This SriovNetwork CR also contains the ‘resourceName’ which is aligned with the ‘resourceName’ of SR-IOV device plugin. One SriovNetwork obj maps to one ‘resoureName’, but one ‘resourceName’ can be shared by different SriovNetwork CRs.

This CR should be managed by cluster admin. Here is an example:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: example-network
namespace: example-namespace
spec:
ipam: |
{
"type": "host-local",
"subnet": "10.56.217.0/24",
"rangeStart": "10.56.217.171",
"rangeEnd": "10.56.217.181",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.56.217.1"
}
vlan: 0
resourceName: intelnics

Chaining CNI metaplugins

It is possible to add additional capabilities to the device configured via the SR-IOV configuring optional metaplugins.

In order to do this, the metaPlugins field must contain the array of one or more additional configurations used to build a network configuration list, as per the following example:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: example-network
namespace: example-namespace
spec:
ipam: |
{
"type": "host-local",
"subnet": "10.56.217.0/24",
"rangeStart": "10.56.217.171",
"rangeEnd": "10.56.217.181",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.56.217.1"
}
vlan: 0
resourceName: intelnics
metaPlugins : |
{
"type": "tuning",
"sysctl": {
"net.core.somaxconn": "500"
}
},
{
"type": "vrf",
"vrfname": "red"
}

OVSNetwork

A custom resource of OVSNetwork could represent the a layer-2 broadcast domain attached to Open vSwitch that works in HW-offloading mode. It is primarily used to generate a NetworkAttachmentDefinition CR with an OVS CNI plugin configuration.

The OVSNetwork CR also contains the resourceName which is aligned with the resourceName of SR-IOV device plugin. One OVSNetwork obj maps to one resourceName, but one resourceName can be shared by different OVSNetwork CRs.

It is expected that resourceName contains name of the resource pool which holds Virtual Functions of a NIC in the switchdev mode. A Physical function of the NIC should be attached to an OVS bridge before any workload which uses OVSNetwork starts.

Example:

apiVersion: sriovnetwork.openshift.io/v1
kind: OVSNetwork
metadata:
name: example-network
namespace: example-namespace
spec:
ipam: |
{
"type": "host-local",
"subnet": "10.56.217.0/24",
"rangeStart": "10.56.217.171",
"rangeEnd": "10.56.217.181",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.56.217.1"
}
vlan: 100
bridge: my-bridge
mtu: 2500
resourceName: switchdevnics

SriovNetworkNodeState

The custom resource to represent the SR-IOV interface states of each host, which should only be managed by the operator itself.

• The ‘spec’ of this CR represents the desired configuration which should be applied to the interfaces and SR-IOV device plugin.
• The ‘status’ contains current states of those PFs (baremetal only), and the states of the VFs. It helps user to discover SR-IOV network hardware on node, or attached VFs in the case of a virtual deployment.

The spec is rendered by sriov-policy-controller, and consumed by sriov-config-daemon. Sriov-config-daemon is responsible for updating the ‘status’ field to reflect the latest status, this information can be used as input to create SriovNetworkNodePolicy CR.

An example of SriovNetworkNodeState CR:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodeState
metadata:
name: worker-node-1
namespace: sriov-network-operator
spec:
interfaces:
- deviceType: vfio-pci
mtu: 1500
numVfs: 4
pciAddress: 0000:86:00.0
status:
interfaces:
- deviceID: "1583"
driver: i40e
mtu: 1500
numVfs: 4
pciAddress: 0000:86:00.0
maxVfs: 64
vendor: "8086"
Vfs:
- deviceID: 154c
driver: vfio-pci
pciAddress: 0000:86:02.0
vendor: "8086"
- deviceID: 154c
driver: vfio-pci
pciAddress: 0000:86:02.1
vendor: "8086"
- deviceID: 154c
driver: vfio-pci
pciAddress: 0000:86:02.2
vendor: "8086"
- deviceID: 154c
driver: vfio-pci
pciAddress: 0000:86:02.3
vendor: "8086"
- deviceID: "1583"
driver: i40e
mtu: 1500
pciAddress: 0000:86:00.1
maxVfs: 64
vendor: "8086"

From this example, in status field, the user can find out there are 2 SRIOV capable NICs on node 'work-node-1'; in spec field, user can…