#Configure Multus multi-NIC networking

This topic describes how to attach ACP distributed storage (Ceph) to dedicated container networks using Multus, so that the Ceph public and cluster traffic can be separated onto different network interfaces (NICs).

#Overview

By default, an ACP Ceph storage cluster uses the host network (provider: host). When you create a storage cluster from the web console, you can also choose Container Network, which places Ceph traffic on a single container network (a single NetworkAttachmentDefinition assigned to the rook-ceph namespace). See Create a Standard Storage Service for the basic single-network option.

Multus multi-NIC networking extends the container-network model so that you can map the two Ceph network roles onto separate networks:

• public network — client traffic between Ceph clients (CSI, applications) and the cluster.
• cluster network — internal traffic between OSDs, such as replication and recovery.

Separating these roles onto different NICs improves isolation and can prevent heavy replication/recovery traffic from competing with client I/O.

#Two management surfaces

ACP provides a legacy and a new web console, which manage Ceph storage through different resources. Configure the network on the resource that manages your cluster:

To determine which resource manages a given cluster, check whether a StorageCluster exists:

kubectl get storagecluster -A

• If a StorageCluster exists for your cluster, it is managed by the new console — configure the network on the StorageCluster (the operator reverts manual edits to the underlying CephCluster).
• If no StorageCluster exists, the cluster is managed directly as a CephCluster by the legacy console — configure the network on the CephCluster.

Both resources expose the same network fields (provider, selectors, addressRanges). The sections below give the manifest for each surface.

The network role to network mapping is declared with selectors:

You can specify only public, or both public and cluster. At least one selector is required when provider is multus.

#Limitations and prerequisites

Before you configure Multus multi-NIC networking, note the following restrictions:

• Creation-time only

  • Multus multi-NIC networking must be configured when the storage cluster is created. Changing the network provider on a running cluster is not supported: Ceph must fail over all monitors to apply a new network model, and the API rejects switching directly from one non-empty provider to another (the provider must first be reverted to an empty value). Treat the network model as a fixed property of the cluster.

• Multus CNI plugin installed

  • The cluster must use Kube-OVN as its network plugin, and the Multus CNI plugin must be installed. This is a one-time, cluster-wide setup. To install it, navigate to Administrator > Marketplace > Cluster Plugins, search for multus, and install Alauda Container Platform Networking for Multus. For details, see Configuring Kube-OVN Network to Support Pod Multi-Network Interfaces.

• Single IP family

  • A Multus network must use a single IP family. IPv4 and IPv6 are both supported, but dual-stack (IPv4 and IPv6 together) is not supported with Multus. To use IPv6, set ipFamily: IPv6 in the network block.
  • This differs from the basic Container Network option in Create a Standard Storage Service, which does not support IPv6. That restriction applies only to that single-network form option; the Multus multi-NIC procedure described here does support IPv6.

#Configure multi-NIC networking at cluster creation

The multi-NIC settings cannot be expressed in the wizard's form fields, but the Create Cluster wizard provides a YAML mode. The recommended flow is to complete the wizard as usual so it auto-populates the manifest, then switch to YAML mode, edit only the network section, and submit from YAML mode. This avoids hand-writing the full manifest.

The manifest shown in YAML mode depends on your console (see Two management surfaces): the new console shows a StorageCluster, the legacy console shows a CephCluster. Edit the network block at the path that matches.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ceph-public-net
  namespace: rook-ceph
spec:
  config: |-
    {
      "cniVersion": "0.3.0",
      "type": "kube-ovn",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "ceph-public-net.rook-ceph.ovn"
    }

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: ceph-public-net
spec:
  protocol: IPv4
  cidrBlock: 192.168.10.0/24
  gateway: 192.168.10.1
  vlan: ceph-public-vlan
  provider: ceph-public-net.rook-ceph.ovn

kubectl get net-attach-def -n rook-ceph

addressRanges:
  public:
    - 10.176.0.0/16
  cluster:
    - 10.176.0.0/16

#Verification

After the storage cluster is created, verify the network configuration from both the ACP and Ceph sides.

#(New console) Check the StorageCluster network settings

If your cluster is managed by a StorageCluster, confirm the desired configuration. A Ceph StorageCluster is always named ceph-cluster in the rook-ceph namespace (both are enforced by admission):

kubectl get storagecluster ceph-cluster -n rook-ceph -o yaml

The output should contain the expected provider and selectors:

spec:
  ceph:
    network:
      provider: multus
      selectors:
        public: rook-ceph/ceph-public-net
        cluster: rook-ceph/ceph-cluster-net

#Check the effective CephCluster network settings

This check applies to both consoles, because the CephCluster is the resource Rook acts on:

kubectl get cephcluster ceph-cluster -n rook-ceph -o jsonpath='{.spec.network}' | jq

The provider and selectors should match what you configured. On the new console, they should also match the managing StorageCluster.

#Check that Ceph Pods are attached to the networks

Confirm that the Ceph daemon Pods (for example, MON and OSD) carry the Multus network attachment annotation and report the expected interfaces:

kubectl get pod -n rook-ceph -l app=rook-ceph-osd \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}{"\n"}{end}'

Each OSD Pod should reference the public (and, if configured, cluster) network.

#Check workload mounts

Create or restart a test workload that mounts a CephFS PVC and an RBD PVC, then verify:

• the Pod starts successfully
• the filesystem can be read and written
• no mount-related errors appear in CSI or workload logs

#Troubleshooting suggestions

If the storage cluster fails to become healthy after enabling Multus, check the following items first:

1. NAD does not exist — the NetworkAttachmentDefinition referenced by a selector is missing from the rook-ceph namespace, or the <namespace>/<nad-name> reference is misspelled.
2. Missing selector — provider: multus is set but no public or cluster selector is defined. At least one selector is required.
3. CIDR detection failures — Rook cannot auto-detect the network range. Specify addressRanges explicitly.
4. IP exhaustion — the IPAM pool backing the NAD (for example, the configured CIDR or address range) has no free addresses for new Ceph Pods.
5. Node cannot reach the underlay — the Ceph CSI driver runs with host networking, so each node (not the workload Pod) must be able to route to the OSD addresses on the underlay network. If mounts hang, verify the node's Kube-OVN underlay routing and that OSD addresses are reachable from the node.

If the cluster cannot be recovered, recreate it with a corrected network configuration. Because the network model cannot be switched on a running cluster, network mistakes are resolved by recreating the storage cluster rather than by patching it in place.