Migrate to Pool-Managed Persistent Disks

Use this page for a Huawei DCS or Huawei Cloud Stack cluster that still keeps platform data, such as /var/cpaas, on disks declared in its machine template. After the migration, these disks are declared in the machine pool, and the upgrade keeps them when it replaces the nodes. Migrate every affected cluster, the global cluster included. For a cluster that Check Cluster Versions sends to technical support, contact technical support before the migration.

WARNING

Complete the migration before you upgrade the cluster. It replaces every node of the cluster with a new VM, one at a time.

Run the commands from the execution host, after ac login, on the global cluster that manages the cluster: the global cluster itself, with global as <cluster-name>, or the primary global cluster for a workload cluster. In a DR environment, migrate each global cluster; for the standby, use its kubeconfig instead of ac login. All objects are in the cpaas-system namespace.

Before You Start

Select the global cluster that runs the commands, and confirm that the printed address ends with /kubernetes/global/:

ac config use-cluster global
ac whoami --show-server

For the standby global cluster in a DR environment, replace the first command with export KUBECONFIG=<standby-kubeconfig>; the second command then prints the VIP of the standby, and every later ac command on this page uses this kubeconfig. Run unset KUBECONFIG when the migration of the standby is complete.

Also confirm that all nodes of the cluster are Ready, and that the cluster has a recent successful etcd backup, as described in Check the etcd Backup.

Then run the following checks on the same cluster.

Check the Provider Requirements

Huawei DCS
Huawei Cloud Stack
  • The cluster runs ACP v4.2.1 or later, as read in Check Cluster Versions, and its VM templates are v4.2.1 or later.
  • Guest tools (vmtools) run inside the VMs.
  • Each node has a fixed IP entry in a DCSIpHostnamePool.
  • You can read the disks of each VM, with their sequenceNum, from the DCS platform or the DCS API (QueryVmInfo).

Then confirm that the provider supports the field:

ac explain dcsiphostnamepools.spec.pool.persistentDisk

Continue when every item above holds and the command describes the field; otherwise, contact technical support.

Check the Rollout Settings

ac -n cpaas-system get clusters.cluster.x-k8s.io <cluster-name> \
  -o jsonpath='{.spec.controlPlaneRef.name}{"\n"}'
ac -n cpaas-system get kubeadmcontrolplane <kcp-name> \
  -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'
ac -n cpaas-system get kubeadmcontrolplane <kcp-name> \
  -o custom-columns='NAME:.metadata.name,TEMPLATE:.spec.machineTemplate.infrastructureRef.name'
ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=<cluster-name> \
  -o custom-columns='NAME:.metadata.name,SURGE:.spec.strategy.rollingUpdate.maxSurge,UNAVAILABLE:.spec.strategy.rollingUpdate.maxUnavailable,TEMPLATE:.spec.template.spec.infrastructureRef.name'

The first command prints the KubeadmControlPlane name, <kcp-name>. The TEMPLATE columns list the machine templates in use; the procedure works on each of them.

ResultAction
replicas is 3 or more and maxSurge is 0, and every MachineDeployment shows SURGE 0 and UNAVAILABLE 1Continue.
replicas is less than 3The cluster cannot be upgraded. Do not migrate it.
replicas is 3 or more and maxSurge is empty or not 0Run the kubeadmcontrolplane command below.
A MachineDeployment shows SURGE empty or not 0, or UNAVAILABLE empty or not 1Run the machinedeployment command below for it.
ac -n cpaas-system patch kubeadmcontrolplane <kcp-name> --type=merge \
  -p '{"spec":{"rolloutStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0}}}}'
ac -n cpaas-system patch machinedeployment <machinedeployment-name> --type=merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}'

Check the Pools for Free Entries

Check each pool that the cluster uses, control-plane and worker pools alike:

Huawei DCS
Huawei Cloud Stack

The last command prints the type and the address of the control-plane load balancer of the cluster:

ac -n cpaas-system get dcsiphostnamepool \
  -o custom-columns='NAME:.metadata.name,IPS:.spec.pool[*].ip,DISKS:.spec.pool[*].persistentDisk[*].path,KEPT_DISKS:.status.persistentDiskStatus[*].ip'
ac -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=<cluster-name> -o custom-columns='NAME:.metadata.name,POOL:.spec.ipHostPoolRef.name,IP:.status.networkConfig.ip'
ac -n cpaas-system get dcscluster <cluster-name> -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneLoadBalancer.host}{"\n"}'

Check each pool in the POOL column. A free entry is an IP address of the pool that no machine of the cluster uses and that is not the load balancer address when the type is internal.

ResultAction
Huawei DCS: the pools have no free entry. Huawei Cloud Stack: each pool in POOL shows AVAILABLE empty or 0Continue.
Huawei DCS: a free entry is not in KEPT_DISKSRemove it from spec.pool with ac -n cpaas-system edit dcsiphostnamepool <pool-name>. Keep the entry of the internal load balancer address.
Huawei DCS: a free entry is in KEPT_DISKSContact technical support. Do not remove the entry.
Huawei Cloud Stack: a pool in POOL shows AVAILABLE greater than 0Run ac -n cpaas-system edit hcsmachineconfigpool <pool-name> and remove the free configurations, the ones in HOSTNAMES that are not in USED, from spec.configs.
Huawei Cloud Stack: the edit is rejected for a free configuration that was used beforeContact technical support.

Migration Procedure

Do each step for the control plane template and for every MachineDeployment template that you listed.

Inspect the Disks and Choose Which to Keep

Huawei DCS
Huawei Cloud Stack

List the VMs of the cluster and the pools:

ac -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=<cluster-name>
ac -n cpaas-system get dcsiphostnamepool

For every disk to keep on each VM, record:

  • From the DCS platform or the VM details of the DCS API (QueryVmInfo): sequenceNum, size (quantityGB), pciType, and the volume URN (volumeUrn).
  • From the entry of the disk in spec.template.spec.vmConfig.dcsMachineDiskSpec[] of the current DCSMachineTemplate: path, format, and datastoreName or datastoreClusterName. The data disk entries follow the old disk order, after the system disk.

A disk can be kept only if it sits at the end of the old disk order. Compute its slot in the pool from its old sequenceNum:

slot = oldSequenceNum - systemDiskCount - newTemplateDataDiskCount - 1

systemDiskCount is 1. newTemplateDataDiskCount is the number of non-system disks that stay in the template after the migration. The slot must be 0 or greater and unique within the IP entry. If a disk gives a negative slot, either also move the disks between it and the end of the old order to the pool, or accept that the disk is deleted with the old VM.

For example, with the old disk order 1 system disk, 2 /var/lib/kubelet, 3 /var/lib/etcd, 4 /var/lib/containerd, 5 /var/cpaas, and a new template that keeps every data disk that is not moved:

Disks to keepOld sequenceNumData disks left in the templateslotCan be kept
/var/cpaas53 (/var/lib/kubelet, /var/lib/etcd, /var/lib/containerd)5 - 1 - 3 - 1 = 0Yes
/var/lib/containerd and /var/cpaas4, 52 (/var/lib/kubelet, /var/lib/etcd)0 and 1Yes
/var/lib/etcd only33 (/var/lib/kubelet, /var/lib/containerd, /var/cpaas)3 - 1 - 3 - 1 = -2No

Update the Machine Template

Huawei DCS
Huawei Cloud Stack

Edit each template in place and keep its name:

ac -n cpaas-system get dcsmachinetemplate <template-name> -o yaml > current-template.yaml

In current-template.yaml:

  • In spec.template.spec.vmConfig.dcsMachineDiskSpec[], keep the system disk and the disks that may be recreated with the VM. Remove every disk to keep, and the disks that you move with it to make it claimable.
  • Keep metadata.name. Remove metadata.resourceVersion, metadata.uid, metadata.creationTimestamp, and metadata.managedFields.

Then apply it with ac apply -f current-template.yaml.

Update the Pool

Huawei DCS
Huawei Cloud Stack

Edit the DCSIpHostnamePool that the template references in spec.template.spec.ipHostPoolRef.name:

ac -n cpaas-system edit dcsiphostnamepool <pool-name>

Under the IP entry of each node, add one persistentDisk entry for each disk to keep, with the slot you computed:

  pool:
  - ip: "<node-ip>"
    mask: "<mask>"
    gateway: "<gateway>"
    dns: "<dns>"
    hostname: "<hostname>"
    machineName: "<machine-name>"
    persistentDisk:
    - slot: <calculated-slot>
      quantityGB: 40
      datastoreClusterName: <datastore-cluster-name>
      path: /var/cpaas
      format: xfs
      pciType: VIRTIO

The existing disk is claimed only when quantityGB (exact size), datastoreName or datastoreClusterName (same storage), and pciType match the live disk. Always set pciType to the value of the live disk; it cannot be changed after you save the entry. path applies on the replacement VM; format and isThin are used only for a new, empty disk.

Confirm the Disks Are Claimed

Before you start the replacement, read the disk states of each pool:

Huawei DCS
Huawei Cloud Stack
ac -n cpaas-system get dcsiphostnamepool <pool-name> \
  -o jsonpath='{range .status.persistentDiskStatus[*]}{.ip}{"\t"}{.slot}{"\t"}{.path}{"\t"}{.phase}{"\t"}{.volumeUrn}{"\t"}{.lastError}{"\n"}{end}'
ResultAction
Every disk to keep is Attached. Huawei DCS: its volumeUrn is the volume URN that you recorded for that disk. Huawei Cloud Stack: it has a volumeIDContinue.
A disk is Attaching or Available, or is missing from the outputRead the states again after one minute; if they do not change, contact technical support.
A disk is in ErrorFix it as described in If a Disk Reports an Error.
Huawei DCS: a disk is Attached with another volumeUrn than the one you recordedDo not start the replacement; contact technical support.
A disk is in Creating or another phaseDo not start the replacement; contact technical support.
A column is emptyRead the pool with -o yaml and handle the status of the disk as in the rows above.

Trigger the Rolling Replacement

Huawei DCS
Huawei Cloud Stack

Start the replacement of the control plane, then of each MachineDeployment:

ac -n cpaas-system patch kubeadmcontrolplane <kcp-name> --type=merge \
  -p '{"spec":{"rolloutAfter":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"}}'
ac -n cpaas-system patch machinedeployment <machinedeployment-name> --type=merge \
  -p '{"spec":{"rolloutAfter":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"}}'

Verify the Migration

Follow the replacement:

ac -n cpaas-system get kubeadmcontrolplane <kcp-name>
ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=<cluster-name>
ac -n cpaas-system get machines.cluster.x-k8s.io -l cluster.x-k8s.io/cluster-name=<cluster-name>

Then read the disk states of each pool with the commands in Confirm the Disks Are Claimed. Each disk goes from Attached on the old VM to Available after it is detached, then to Attached on the replacement VM; Detaching and Attaching appear in between. The migration is complete when the KubeadmControlPlane shows REPLICAS equal to DESIRED, the KubeadmControlPlane and every MachineDeployment show UPDATED equal to REPLICAS and UNAVAILABLE at 0, every Machine is Running, and every disk is Attached.

If a Disk Reports an Error

When a disk is in phase: Error, read its lastError with the command in Confirm the Disks Are Claimed. Fix the cause, and the provider retries and the replacement continues. If the cause cannot be fixed, contact technical support.

Huawei DCS
Huawei Cloud Stack

A claim fails when quantityGB, the datastore, or pciType in the pool entry differs from the live disk. For quantityGB or the datastore, correct the pool entry; the provider retries on its own. pciType cannot be changed after the entry is saved; if it differs, contact technical support.

Limitations

  • Only disks declared in the pool are kept. Every other disk of the machine template is deleted with the old VM.
  • Do not declare the same disk in both the machine template and the pool.

When every affected cluster is migrated, return to Prepare for the Upgrade.