Reconciliation loop

Modules

Each time a new Module is created, we need to find to which nodes it applies.

  1. A first filtering is performed using the .spec.selector field, then we go through the module’s kernel mappings to find a container image that matches the node’s kernel.

  2. We end up with a certain number of (kernel, image) pairs; for each of these pairs, there should be a DaemonSet.

  3. We first look for a DaemonSet that would already be targeting the same kernel and DriverContainer image (that data is stored in the DaemonSet’s labels).

  4. If there is already such a DaemonSet, we patch it, if needed.

  5. If there is not already a matching DaemonSet, we create it and set the Module as owner.

  6. When a Module is deleted, we have nothing to do: because we set it as owner of all DaemonSets, Kubernetes garbage collection will take care of deleting them.

  7. We watch Module, owned DaemonSet and build objects as well as nodes to make sure that we are not missing any change in the cluster.

Modules reconciliation

DRA

A separate DRA reconciler watches Module resources that have .spec.dra set.

  1. When a Module has .spec.dra configured, the DRA reconciler creates a DRA driver DaemonSet targeting nodes where the kernel module is loaded and that carry the kmm.node.kubernetes.io/<namespace>.<modulename>.dra-target label. The reconciler puts that label on every schedulable node the Module selects and takes it off a node that is no longer schedulable, so the DRA driver DaemonSet is evicted from a draining node. A node that stops matching .spec.selector is not visited here and keeps the label until spec.dra is removed or the Module is deleted. A Module with no .spec.moduleLoader has no kernel module to unload, so its DaemonSet targets .spec.selector directly and does not select on the label.

  2. It also creates and manages cluster-scoped DeviceClass resources as declared in .spec.dra.deviceClasses. DeviceClasses are tracked via labels (kmm.node.kubernetes.io/module.name and kmm.node.kubernetes.io/module.namespace) since they are cluster-scoped while Modules are namespaced.

  3. When spec.dra is removed or the Module is deleted, the reconciler deletes all associated DRA DaemonSet and DeviceClass resources.

  4. During ordered upgrades, a new DRA DaemonSet is created for the new module version. Once the old-version DaemonSet is no longer scheduled on any node, it is garbage-collected.

  5. We watch Module, owned DaemonSet, DeviceClass and nodes, since the dra-target label is decided from a node's own labels and taints.