MySQL-PXC Removed in Alauda Data Services Relational Database E1 v4.3.0
If you have MySQL-PXC instances, they will become unmanaged after upgrading to ACP v4.3.0 or later (when the MySQL operator is also upgraded). See MySQL Operator Upgrade Guide for migration instructions.
Complete these steps before you upgrade. Prepare again for every upgrade, including a patch upgrade of the ACP version (Distribution Version) and an upgrade of global before every workload cluster has caught up.
In a DR environment with Alauda Container Platform Global Cluster Disaster Recovery, do every step that names global on both global clusters. If your clusters run on more than one provider, complete the steps for each of them, including the provider of the global cluster.
Run every upgrade command from one execution host: a Linux machine, x86_64 or aarch64, that can reach ACP. Do not use a node of global or of any other cluster; the upgrade does not need one. On Alauda OS, the upgrade replaces the nodes, and files on a replaced node, such as the Core Package and the contents of /root and /home, are lost. The few steps that run on a node say so; return to the execution host after each of them.
The execution host needs:
Access to the platform address, and to the VIP and registry of each global cluster.
The target Core Package, downloaded as described in Download the Core Package, extracted on this host, and kept until every cluster is upgraded.
ACP CLI (ac). The Core Package contains it for both architectures, at installer/res/<arch>/ac, where <arch> is amd64 or arm64. Copy the one for this host to a directory in your PATH; for example, from the installer/ directory on an x86_64 host, run sudo cp res/amd64/ac /usr/local/bin/ac.
kubectl, and Violet v4.2.13 or later (violet version).
A platform account with the platform administrator role.
In DR, a kubeconfig of the standby global cluster that points to its VIP, <standby-kubeconfig>. For example, copy /etc/kubernetes/admin.conf, with any certificate files it references, from a control-plane node of the standby, and set its server to the VIP if it points elsewhere.
Log in once with the platform administrator account, with KUBECONFIG unset:
ac login https://<platform-address> --name <session>ac config use-cluster <cluster-name>ac whoami --show-serverac config use-cluster global
The session is saved in ~/.kube/config, and ac commands run on global, the primary in DR. For a workload cluster, run ac config use-cluster <cluster-name> first and ac config use-cluster global afterwards; ac whoami --show-server then ends with /kubernetes/<cluster-name>/.
upgrade.sh uses kubectl with this session. Run ac config use-cluster global right before the script.
For the standby, run export KUBECONFIG=<standby-kubeconfig> first, confirm that ac whoami --show-server prints the standby VIP, and run unset KUBECONFIG afterwards.
Applies to: every cluster registered on global, global included, imported clusters included. In DR, run it on each global cluster; on the standby, read only its own version.
On global, list the ACP version (the CURRENT column of the first list), the Kubernetes version, and the infrastructure provider of every cluster. In the last list, DCSCluster, BaremetalCluster, HCSCluster, and VSphereCluster mark an Alauda OS cluster, UserProvisionedCluster a cluster that does not run Alauda OS, and <none> an imported cluster:
ac config use-cluster globalac whoami --show-serverac -n cpaas-system get cvshac -n cpaas-system get clusters.clusterregistry.k8s.io -o custom-columns='NAME:.metadata.name,KUBERNETES:.status.version'ac -n cpaas-system get clusters.cluster.x-k8s.io -o custom-columns='NAME:.metadata.name,INFRASTRUCTURE:.spec.infrastructureRef.kind'
If cvsh reports the server doesn't have a resource type, or does not list a cluster, take the ACP version of that cluster from VERSION:
ac get clustermodule -o custom-columns='NAME:.metadata.name,VERSION:.spec.version'
In DR, read the version of the standby, from CURRENT, or from VERSION when cvsh is not available:
ac --kubeconfig <standby-kubeconfig> -n cpaas-system get cvsh globalac --kubeconfig <standby-kubeconfig> get clustermodule global -o custom-columns='NAME:.metadata.name,VERSION:.spec.version'
Compare each cluster with the table. Continue when every cluster matches a row whose action is Continue; otherwise, take the action of each row that matches.
Result for a cluster
Action
ACP 4.2, 4.3, or 4.4 (on Alauda OS: ACP v4.3.2 or later); and, for a cluster created by ACP, a Kubernetes version within the Compatible Versions of the target in the Kubernetes Support Matrix
Continue. The preflight check VersionUpgradePath confirms the exact patch version.
Already the target minor version: a patch upgrade, or global upgraded again
Continue.
ACP 4.1 or earlier
Contact technical support.
On Alauda OS, an ACP version earlier than v4.3.2
Contact technical support.
Created by ACP, with a Kubernetes version outside the Compatible Versions of the target
Upgrade the cluster on the current global first, as described in the documentation of its current ACP version.
No version is shown for the cluster
Contact technical support.
#Check the Registry, Modules, and etcd Backup of global
Applies to: every global cluster. In DR, run each block on the standby too, as the comments on its first and last lines show: start with export KUBECONFIG=<standby-kubeconfig> and finish with unset KUBECONFIG.
Save this script as check-global-common.sh:
#!/usr/bin/env bashac whoami --show-serverecho "== Registry"ac get productbase base -o jsonpath='external={.spec.registry.external}{"\n"}'echo "== Modules"ac get moduleinfo -l cpaas.io/cluster-name=global \ -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,PHASE:.status.phase'echo "== etcd backup records"ac get etcdbackupconfiguration etcd-backup-default \ -o jsonpath='{range .status.records[*]}{.backupTimestamp}{"\t"}{.result}{"\n"}{end}'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-common.sh# on the standby: unset KUBECONFIG
external= with an empty value or false is the platform built-in Registry. With it, on each control-plane node of global, run the line for the OS of the node:
The etcd backup record with the latest time shows Success
Continue.
The etcd command reports NotFound or the server doesn't have a resource type
Install Alauda Container Platform Cluster Enhancement (shown as Alauda Container Platform Cluster Enhancer on a cluster that runs ACP 4.4 or earlier) on the cluster, as described in etcd Backup and Restore, and run the command again after its first backup. If you cannot install it, contact technical support.
The etcd backup record with the latest time does not show Success, or there is no record
Applies to: every global cluster. In DR, run each block on the standby too, as the comments on its first and last lines show: start with export KUBECONFIG=<standby-kubeconfig> and finish with unset KUBECONFIG.
Run each block on the execution host unless it says that it runs on a node. Each table has a Continue row for each part of the output: continue when every part matches its Continue row; otherwise, take the action of each row that matches. Check the Staged Images checks the images after you stage them.
Use the Non-Alauda OS tab when the nodes of global do not run Alauda OS; otherwise, use the tab of the infrastructure provider of global, shown in INFRASTRUCTURE in Check Cluster Versions.
WARNING
On Alauda OS, data on a node's own disks, such as hostPath volumes and local PersistentVolumes, is not kept when the node is replaced, unless it is on a pool-managed persistent disk. Move it to CSI storage before the upgrade.
Non-Alauda OS
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
On each node of global:
uname -rstat -fc %T /sys/fs/cgroup/
Continue when the kernel is 5.8 or later and within Supported OS and Kernel Versions, and the second command prints cgroup2fs; otherwise, update the OS of the node before the upgrade, and if you cannot, contact technical support. The update restarts the node.
Save this script as check-global-dcs.sh:
#!/usr/bin/env bashac whoami --show-serverecho "== MachineDeployments"ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,UNAVAILABLE:.spec.strategy.rollingUpdate.maxUnavailable'echo "== Machines, IP pools, and the type and address of the control-plane load balancer"ac -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,POOL:.spec.ipHostPoolRef.name,IP:.status.networkConfig.ip'ac -n cpaas-system get dcsiphostnamepool \ -o custom-columns='NAME:.metadata.name,IPS:.spec.pool[*].ip,DISKS:.spec.pool[*].persistentDisk[*].slot,KEPT_DISKS:.status.persistentDiskStatus[*].ip'ac -n cpaas-system get dcscluster global -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneLoadBalancer.host}{"\n"}'echo "== Secrets that the DCSCluster references"ac -n cpaas-system get secret $(ac -n cpaas-system get dcscluster global -o jsonpath='{.spec.credentialSecretRef.name} {.spec.encryptionProviderConfigRef.name}')echo "== Local, hostPath, and TopoLVM PersistentVolumes with a claim"ac get pv --no-headers \ -o custom-columns='NAME:.metadata.name,CLAIM_NAMESPACE:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,LOCAL:.spec.local.path,HOSTPATH:.spec.hostPath.path,CSI:.spec.csi.driver' \ | awk '$2 !~ /^(<none>|kube-system|kube-public|kube-node-lease|cpaas-system)$/ && ($4 != "<none>" || $5 != "<none>" || $6 ~ /^topolvm\.(io|cybozu\.com)$/)'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-dcs.sh# on the standby: unset KUBECONFIG
A free entry is an IP address of a pool in POOL whose DISKS lists slot numbers, that no machine uses, and that is not the load balancer address when the type is internal.
Result
Action
Every MachineDeployment shows UNAVAILABLE1
Continue.
A MachineDeployment shows UNAVAILABLE empty, <none>, or not 1
The pools in POOL whose DISKS lists slot numbers have no free entry
Continue.
A free entry is not in KEPT_DISKS
Remove it from spec.pool with ac -n cpaas-system edit dcsiphostnamepool <pool-name>. Keep the entry of the internal load balancer address.
A free entry is in KEPT_DISKS
Contact technical support. Do not remove the entry.
Each Secret is listed
Continue.
A Secret is reported NotFound
Contact technical support.
Nothing is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Continue.
A PersistentVolume is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume.
Run the next script only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
Save this script as check-global-dcs-no-preflight.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Rollout settings"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,SURGE:.spec.strategy.rollingUpdate.maxSurge'echo "== Machine templates in use"ac -n cpaas-system get dcsmachinetemplate \ $(ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{.spec.machineTemplate.infrastructureRef.name}') \ $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.infrastructureRef.name}') \ -o custom-columns='NAME:.metadata.name,TEMPLATE_DISKS:.spec.template.spec.vmConfig.dcsMachineDiskSpec[*].path'echo "== Machines"ac -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,VM_TEMPLATE:.spec.vmTemplateName,READY:.status.ready'echo "== Kubelet patch files: the encoding, then the content or a Secret name and key"echo "${KCP}:"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{range .spec.kubeadmConfigSpec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'for t in $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.bootstrap.configRef.name}'); do echo "${t}:" ac -n cpaas-system get kubeadmconfigtemplate "${t}" -o jsonpath='{range .spec.template.spec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'doneecho "== Pods with hostPath volumes"ac get pods -A --no-headers \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[*].kind,MIRROR:.metadata.annotations.kubernetes\.io/config\.mirror,PHASE:.status.phase,HOSTPATH:.spec.volumes[*].hostPath.path' \ | awk '$1 !~ /^(kube-system|kube-public|kube-node-lease|cpaas-system)$/ && $3 !~ /DaemonSet/ && $4 == "<none>" && $5 !~ /^(Succeeded|Failed)$/ && $6 != "<none>" { n = split($6, p, ","); keep = 0 for (i = 1; i <= n; i++) if (p[i] !~ /^\/(dev|proc|sys|run|var\/run|var\/lib\/kubelet)(\/|$)/) keep = 1 if (keep) print }'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-dcs-no-preflight.sh# on the standby: unset KUBECONFIG
Result
Action
replicas is 3 or more and maxSurge is 0, and every MachineDeployment shows SURGE0
Continue.
replicas is less than 3
The cluster cannot be upgraded.
replicas is 3 or more and maxSurge is empty or not 0, or a MachineDeployment shows SURGE empty, <none>, or not 0
Run the script again later. If the machine still does not show true, contact technical support.
Every machine shows the same VM_TEMPLATE
Continue.
The machines show different VM_TEMPLATE values
Contact technical support.
Each kubelet patch file shows encoding=[], then its content, or a Secret name and a key for which the command in Kubelet Patch File prints the content
Continue.
No encoding= line follows a name under == Kubelet patch files, a file shows a value inside encoding=[], such as encoding=[base64], or no content, or its Secret or key does not exist
Contact technical support.
Nothing is listed under == Pods with hostPath volumes
Continue.
A Pod is listed under == Pods with hostPath volumes
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume. A hostPath under a persistent disk declared for the node in DCSIpHostnamePool does not count; no action is needed for it.
Save this script as check-global-baremetal.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Control plane and MachineDeployments"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='replicas={.spec.replicas}{"\n"}'ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,UNAVAILABLE:.spec.strategy.rollingUpdate.maxUnavailable'echo "== Host pools and hosts"ac -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io \ -o custom-columns='NAME:.metadata.name,CLUSTER:.spec.clusterName,HOSTS:.spec.machineInventories[*].name'ac -n cpaas-system get machineinventories.elemental.cattle.io \ -o custom-columns='NAME:.metadata.name,OS:.metadata.labels.os\.baremetal\.alauda\.io/id'echo "== Local, hostPath, and TopoLVM PersistentVolumes with a claim"ac get pv --no-headers \ -o custom-columns='NAME:.metadata.name,CLAIM_NAMESPACE:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,LOCAL:.spec.local.path,HOSTPATH:.spec.hostPath.path,CSI:.spec.csi.driver' \ | awk '$2 !~ /^(<none>|kube-system|kube-public|kube-node-lease|cpaas-system)$/ && ($4 != "<none>" || $5 != "<none>" || $6 ~ /^topolvm\.(io|cybozu\.com)$/)'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-baremetal.sh# on the standby: unset KUBECONFIG
The hosts of global are the ones in HOSTS of the pools whose CLUSTER is global, spare hosts included. On each of these hosts whose OS is alaudaos.m.55, empty, or <none> (Alauda OS M55):
For Alauda OS KU hosts, the free space of /persist must be larger than the ROOT partition (20 GiB) plus 2 GiB. Every host must also reach the registry of its cluster.
WARNING
A node pool with a single host has no worker while that host is reinstalled. Make sure its workloads can run on other nodes, or accept the interruption.
Result
Action
Every MachineDeployment shows UNAVAILABLE1
Continue.
A MachineDeployment shows UNAVAILABLE empty, <none>, or not 1
Nothing is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Continue.
A PersistentVolume is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume.
COS_STATE is larger than 8 GiB, Free (estimated) is at least 4 GiB, and no snapshot shows no in default and active and update-in-progress=yes in userdata
Continue.
Free (estimated) is less than 4 GiB, or a snapshot shows no in default and active and update-in-progress=yes in userdata
Before the upgrade, install Alauda OS on the host again from an ISO built as described in Create the Registration and the ISO, which has a larger partition. Installing the host again replaces its node.
Run the next script only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
Save this script as check-global-baremetal-no-preflight.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Rollout settings"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,SURGE:.spec.strategy.rollingUpdate.maxSurge'echo "== Machines"ac -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,PHASE:.status.phase'echo "== Kubelet patch files: the encoding, then the content or a Secret name and key"echo "${KCP}:"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{range .spec.kubeadmConfigSpec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'for t in $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.bootstrap.configRef.name}'); do echo "${t}:" ac -n cpaas-system get kubeadmconfigtemplate "${t}" -o jsonpath='{range .spec.template.spec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'doneecho "== Pods with hostPath volumes"ac get pods -A --no-headers \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[*].kind,MIRROR:.metadata.annotations.kubernetes\.io/config\.mirror,PHASE:.status.phase,HOSTPATH:.spec.volumes[*].hostPath.path' \ | awk '$1 !~ /^(kube-system|kube-public|kube-node-lease|cpaas-system)$/ && $3 !~ /DaemonSet/ && $4 == "<none>" && $5 !~ /^(Succeeded|Failed)$/ && $6 != "<none>" { n = split($6, p, ","); keep = 0 for (i = 1; i <= n; i++) if (p[i] !~ /^\/(dev|proc|sys|run|var\/run|var\/lib\/kubelet)(\/|$)/) keep = 1 if (keep) print }'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-baremetal-no-preflight.sh# on the standby: unset KUBECONFIG
Result
Action
replicas is 3 or more and maxSurge is 0, and every MachineDeployment shows SURGE0
Continue.
replicas is less than 3
The cluster cannot be upgraded.
replicas is 3 or more and maxSurge is empty or not 0, or a MachineDeployment shows SURGE empty, <none>, or not 0
A machine shows PHASE empty, <none>, or not Running
Run the script again later. If the machine still does not show Running, contact technical support.
Each kubelet patch file shows encoding=[], then its content, or a Secret name and a key for which the command in Kubelet Patch File prints the content
Continue.
No encoding= line follows a name under == Kubelet patch files, a file shows a value inside encoding=[], such as encoding=[base64], or no content, or its Secret or key does not exist
Contact technical support.
Nothing is listed under == Pods with hostPath volumes
Continue.
A Pod is listed under == Pods with hostPath volumes
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume.
Save this script as check-global-hcs.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Pools of the machine templates in use"ac -n cpaas-system get hcsmachinetemplate \ $(ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{.spec.machineTemplate.infrastructureRef.name}') \ $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.infrastructureRef.name}') \ -o custom-columns='NAME:.metadata.name,POOL:.spec.template.spec.configPoolRef.name'ac -n cpaas-system get hcsmachineconfigpool \ -o custom-columns='NAME:.metadata.name,AVAILABLE:.status.availableCount,DISKS:.spec.configs[*].persistentDisks[*].slot,HOSTNAMES:.spec.configs[*].hostname,USED:.status.usedConfigs[*].hostname'echo "== Providers"ac get moduleinfo -l cpaas.io/cluster-name=global \ -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,PHASE:.status.phase' | grep -E '^MODULE|cluster-api-provider'echo "== Local, hostPath, and TopoLVM PersistentVolumes with a claim"ac get pv --no-headers \ -o custom-columns='NAME:.metadata.name,CLAIM_NAMESPACE:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,LOCAL:.spec.local.path,HOSTPATH:.spec.hostPath.path,CSI:.spec.csi.driver' \ | awk '$2 !~ /^(<none>|kube-system|kube-public|kube-node-lease|cpaas-system)$/ && ($4 != "<none>" || $5 != "<none>" || $6 ~ /^topolvm\.(io|cybozu\.com)$/)'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-hcs.sh# on the standby: unset KUBECONFIG
Result
Action
Every pool in POOL whose DISKS lists slot numbers shows AVAILABLE empty, <none>, or 0
Continue.
Such a pool shows AVAILABLE greater than 0
Remove its free configurations, the ones in HOSTNAMES that are not in USED, from spec.configs with ac -n cpaas-system edit hcsmachineconfigpool <pool-name>.
The edit is rejected for a free configuration that was used before
Keep the configuration: its persistent disk is still tracked. Scale up a node group that uses the pool so that a new node takes it, and keep the control plane at an odd number of nodes. Do not delete EVS disks.
cluster-api-provider-kubeadm and cluster-api-provider-hcs show PHASERunning
Continue.
A provider is missing, or shows another PHASE
Contact technical support.
Nothing is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Continue.
A PersistentVolume is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume.
Run the next script only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
Save this script as check-global-hcs-no-preflight.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Rollout settings"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,SURGE:.spec.strategy.rollingUpdate.maxSurge,UNAVAILABLE:.spec.strategy.rollingUpdate.maxUnavailable'echo "== Machine templates in use"ac -n cpaas-system get hcsmachinetemplate \ $(ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{.spec.machineTemplate.infrastructureRef.name}') \ $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.infrastructureRef.name}') \ -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.imageName,DATA_VOLUMES:.spec.template.spec.dataVolumes[*].mountPath'echo "== Kubelet patch files: the encoding, then the content or a Secret name and key"echo "${KCP}:"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{range .spec.kubeadmConfigSpec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'for t in $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.bootstrap.configRef.name}'); do echo "${t}:" ac -n cpaas-system get kubeadmconfigtemplate "${t}" -o jsonpath='{range .spec.template.spec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'doneecho "== Pods with hostPath volumes"ac get pods -A --no-headers \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[*].kind,MIRROR:.metadata.annotations.kubernetes\.io/config\.mirror,PHASE:.status.phase,HOSTPATH:.spec.volumes[*].hostPath.path' \ | awk '$1 !~ /^(kube-system|kube-public|kube-node-lease|cpaas-system)$/ && $3 !~ /DaemonSet/ && $4 == "<none>" && $5 !~ /^(Succeeded|Failed)$/ && $6 != "<none>" { n = split($6, p, ","); keep = 0 for (i = 1; i <= n; i++) if (p[i] !~ /^\/(dev|proc|sys|run|var\/run|var\/lib\/kubelet)(\/|$)/) keep = 1 if (keep) print }'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-hcs-no-preflight.sh# on the standby: unset KUBECONFIG
Result
Action
replicas is 3 or more and maxSurge is 0, and every MachineDeployment shows SURGE0 and UNAVAILABLE1
Continue.
replicas is less than 3
The cluster cannot be upgraded.
replicas is 3 or more and maxSurge is empty or not 0, or a MachineDeployment shows SURGE empty, <none>, or not 0, or UNAVAILABLE empty, <none>, or not 1
Each kubelet patch file shows encoding=[], then its content, or a Secret name and a key for which the command in Kubelet Patch File prints the content
Continue.
No encoding= line follows a name under == Kubelet patch files, a file shows a value inside encoding=[], such as encoding=[base64], or no content, or its Secret or key does not exist
Contact technical support.
Nothing is listed under == Pods with hostPath volumes
Continue.
A Pod is listed under == Pods with hostPath volumes
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume. A hostPath under a persistent disk declared for the node in HCSMachineConfigPool does not count; no action is needed for it.
Do not restructure the disk lists of the machine config pools as part of an upgrade.
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-vsphere.sh# on the standby: unset KUBECONFIG
Result
Action
Every pool whose CLUSTER is global shows READYTrue
Continue.
Such a pool does not show True
Contact technical support.
cluster-api-provider-kubeadm and cluster-api-provider-vsphere show PHASERunning
Continue.
A provider is missing, or shows another PHASE
Contact technical support.
Nothing is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Continue.
A PersistentVolume is listed under == Local, hostPath, and TopoLVM PersistentVolumes with a claim
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume.
Run the next script only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
Save this script as check-global-vsphere-no-preflight.sh:
#!/usr/bin/env bashac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.spec.controlPlaneRef.name}')echo "== Rollout settings"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='replicas={.spec.replicas} maxSurge={.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}'ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global \ -o custom-columns='NAME:.metadata.name,SURGE:.spec.strategy.rollingUpdate.maxSurge,UNAVAILABLE:.spec.strategy.rollingUpdate.maxUnavailable'echo "== Machine templates in use"ac -n cpaas-system get vspheremachinetemplate \ $(ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{.spec.machineTemplate.infrastructureRef.name}') \ $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.infrastructureRef.name}') \ -o custom-columns='NAME:.metadata.name,VM_TEMPLATE:.spec.template.spec.template'echo "== Kubelet patch files: the encoding, then the content or a Secret name and key"echo "${KCP}:"ac -n cpaas-system get kubeadmcontrolplane "${KCP}" -o jsonpath='{range .spec.kubeadmConfigSpec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'for t in $(ac -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global -o jsonpath='{.items[*].spec.template.spec.bootstrap.configRef.name}'); do echo "${t}:" ac -n cpaas-system get kubeadmconfigtemplate "${t}" -o jsonpath='{range .spec.template.spec.files[?(@.path=="/etc/kubernetes/patches/kubeletconfiguration0+strategic.json")]}{"encoding=["}{.encoding}{"] "}{.content}{.contentFrom.secret.name}{" "}{.contentFrom.secret.key}{"\n"}{end}'doneecho "== Pods with hostPath volumes"ac get pods -A --no-headers \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[*].kind,MIRROR:.metadata.annotations.kubernetes\.io/config\.mirror,PHASE:.status.phase,HOSTPATH:.spec.volumes[*].hostPath.path' \ | awk '$1 !~ /^(kube-system|kube-public|kube-node-lease|cpaas-system)$/ && $3 !~ /DaemonSet/ && $4 == "<none>" && $5 !~ /^(Succeeded|Failed)$/ && $6 != "<none>" { n = split($6, p, ","); keep = 0 for (i = 1; i <= n; i++) if (p[i] !~ /^\/(dev|proc|sys|run|var\/run|var\/lib\/kubelet)(\/|$)/) keep = 1 if (keep) print }'
Then run it:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>bash check-global-vsphere-no-preflight.sh# on the standby: unset KUBECONFIG
Result
Action
replicas is 3 or more and maxSurge is 0, and every MachineDeployment shows SURGE0 and UNAVAILABLE1
Continue.
replicas is less than 3
The cluster cannot be upgraded.
replicas is 3 or more and maxSurge is empty or not 0, or a MachineDeployment shows SURGE empty, <none>, or not 0, or UNAVAILABLE empty, <none>, or not 1
The machine templates show different VM_TEMPLATE values
Contact technical support.
Each kubelet patch file shows encoding=[], then its content, or a Secret name and a key for which the command in Kubelet Patch File prints the content
Continue.
No encoding= line follows a name under == Kubelet patch files, a file shows a value inside encoding=[], such as encoding=[base64], or no content, or its Secret or key does not exist
Contact technical support.
Nothing is listed under == Pods with hostPath volumes
Continue.
A Pod is listed under == Pods with hostPath volumes
Move its data to CSI storage before the upgrade. If you have confirmed that the data is safe or can be lost, exempt the volume. A hostPath under a persistent disk declared for the node in VSphereMachineConfigPool does not count; no action is needed for it.
Use this when a check table or another page points here. Set <cluster-name> to the cluster of the check: global, or a workload cluster. Run the kubeadmcontrolplane patch when replicas is 3 or more and maxSurge is empty or not 0; when replicas is less than 3, the cluster cannot be upgraded. Run the machinedeployment patch for each MachineDeployment whose maxSurge is empty, <none>, or not 0, or whose maxUnavailable is empty, <none>, or not 1.
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverKCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io <cluster-name> -o jsonpath='{.spec.controlPlaneRef.name}')# The control plane, only when replicas is 3 or moreac -n cpaas-system patch kubeadmcontrolplane "${KCP}" --type merge \ -p '{"spec":{"rolloutStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0}}}}'# Each MachineDeployment to correctac -n cpaas-system patch machinedeployment <md-name> --type merge \ -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}'# on the standby: unset KUBECONFIG
Use this when a check table or another page points here. When a check shows a Secret name and a key for the file /etc/kubernetes/patches/kubeletconfiguration0+strategic.json, print the content from the Secret. Escape each . in <secret-key> as \.:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get secret <secret-name> -o jsonpath='{.data.<secret-key>}' | base64 -d# on the standby: unset KUBECONFIG
On Kubernetes 1.35 or later, the file needs this setting. Do not add it for Kubernetes 1.34 or earlier. In the Kubernetes upgrade, the platform adds it to the existing files.
ac config use-cluster <cluster-name> # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac get etcdbackupconfiguration etcd-backup-default \ -o jsonpath='{range .status.records[*]}{.backupTimestamp}{"\t"}{.result}{"\n"}{end}'ac config use-cluster global # on the standby, instead: unset KUBECONFIG
Result
Action
The record with the latest time shows Success
Continue.
The command reports NotFound or the server doesn't have a resource type
Install Alauda Container Platform Cluster Enhancement (shown as Alauda Container Platform Cluster Enhancer on a cluster that runs ACP 4.4 or earlier) on the cluster, as described in etcd Backup and Restore, and run the command again after its first backup. If you cannot install it, contact technical support.
The record with the latest time does not show Success, or there is no record
Contact technical support.
#Check Node Readiness for Kubernetes 1.35 or Later
Use this when a check table or another page points here. On a non-Alauda OS cluster created by ACP, run the node check of the Non-Alauda OS tab, on every node of the cluster:
Applies to: Alauda OS clusters. Skip it for non-Alauda OS clusters, and go to Download the Packages.
Select the current version of each cluster, and prepare every version that the calculator lists for all paths, in the Alauda OS variant that each cluster runs today:
Step
ACP version
Kubernetes version
etcd
Current
v4.4.1
v1.35.6-acp.9
v3.6.13-acp.2
Target
v4.5.0
v1.36.4-acp.3
v3.6.13-acp.5
Prepare the Alauda OS image of every Intermediate and Target row, using the same Alauda OS variant that the cluster runs today.
The Kubernetes version and the etcd tag of every ACP version are listed in Version Mapping.
#Download the Core Package and the Alauda OS Images
From the Customer Portal, download the Core Package of the target ACP version for your architecture, and extract it on the execution host. On Alauda OS, also download:
The Core Package of the ACP release of each intermediate Kubernetes version. You use it only to publish its images.
From the OS download page, the image of the target version and of each intermediate Kubernetes version, for the architecture and OS variant of your nodes: qcow2 for Huawei DCS and Huawei Cloud Stack, ova for VMware vSphere, and the archive of an OCI image layout for Bare Metal.
Export the Extensions installed on global to apps.yaml. Then let the Customer Portal check the file and download the target-version packages of the installed Extensions, including the infrastructure providers and the Cluster Bootstrap Provider. Set <arch> to amd64, arm64, or hybrid to match the Core Package, and <target-version> to the target ACP version, such as 4.5.0.
violet ac uses the address and access token of the Customer Portal; violet list uses the platform address and an ACP API token. See Addresses and Credentials.
Run violet ac on a machine that has access to the Customer Portal. If the execution host has none, run violet list on the execution host, copy apps.yaml to such a machine, run violet ac login and violet ac import-yaml there, and copy the downloaded packages to the execution host. If a command fails, see Troubleshooting.
In a DR environment, do the same for the standby, into its own file. The two global clusters can run different Extensions; for example, Global Cluster Disaster Recovery runs only on the standby:
#Download the Upgrade Scenario for ACP 4.3 or Earlier
Only when the source ACP version is 4.3 or earlier. With the Customer Portal login of the previous step, download the Upgrade from ACP 4.3 or earlier scenario:
violet ac scenarios --arch=<arch> --platformVersion=v<target-version>violet ac download-app --arch=<arch> --platformVersion=v<target-version> --scenario="Upgrade from ACP 4.3 or earlier"
Only when the source ACP version is 4.3 or earlier. Copy every package of the upgrade scenario into plugins/, then list the directory and compare it with the downloaded packages.
Read the registry type of global: the second value is empty or false for the platform built-in Registry, and true for an external registry. Every Aligned Extension installed on a cluster needs its target-version package before that cluster's upgrade; otherwise the upgrade stops until you publish it.
ac config use-cluster globalac whoami --show-serverac get productbase base -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'
Platform built-in Registry
External registry
ac config use-cluster globalac whoami --show-serverbash upgrade.sh --only-sync-image
upgrade.sh does not upload to an external registry. Run both upload modes; both must succeed. Do not export the registry username or password: bash upgrade.sh --preflight stops while either is set in the environment.
On Alauda OS, when the path has an intermediate Kubernetes version, publish the images of the Core Package of each intermediate Kubernetes version, from its installer/ directory, against the same global cluster. Do not request an upgrade to that release. Skip this step when there is no intermediate Kubernetes version.
Platform built-in Registry
External registry
ac config use-cluster globalac whoami --show-serverbash upgrade.sh --only-sync-image
Publish the packages downloaded for apps.yaml in Download the Other Extension Packages with violet push. The packages for the standby follow in the next step.
Packages
Publish to
When
Cluster plugins, including the providers
global
Before the global upgrade
Operators
global and each workload cluster that runs the operator
# A cluster plugin: the platform distributes itviolet push <cluster-plugin-package>.tgz --platform-address "https://<platform-address>" --platform-token "${PLATFORM_TOKEN}" \ --dest-repo "${REGISTRY_ADDRESS}" --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}"# An operator: global and every workload cluster that runs itviolet push <operator-package>.tgz --platform-address "https://<platform-address>" --platform-token "${PLATFORM_TOKEN}" \ --clusters "global,<cluster-name-1>,<cluster-name-2>" \ --dest-repo "${REGISTRY_ADDRESS}" --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}"unset REGISTRY_USERNAME REGISTRY_PASSWORD
Violet pushes the images only to --dest-repo. When a workload cluster uses a registry other than that of global, push each of its operators again, with --clusters "<cluster-name>" and --dest-repo and the credentials of that registry.
In a DR environment only. Select the standby for the rest of this step, and read its registry type, as on the primary:
export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac get productbase base -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'
Platform built-in Registry
External registry
Publish the images to the registry of the standby: run the command from the installer/ directory of the target Core Package, and again from that of each intermediate Core Package. The script uses the registry of the cluster in KUBECONFIG.
bash upgrade.sh --only-sync-image
Then push the packages downloaded for standby-apps.yaml in Download the Other Extension Packages. Set <registry-port> to the port of the registry address printed above. Always set --platform-address and --dest-repo; without them, the packages go to the primary.
upgrade.sh does not upload to an external registry. When the standby uses the same registry endpoint as the primary, the images are already there. When it uses its own endpoint, run the external registry commands of Publish Core and the Staged Packages and Publish the Intermediate Core Images again, with the address and credentials of that endpoint.
Then push the packages downloaded for standby-apps.yaml. Always set --platform-address and --dest-repo; without them, the packages go to the primary. Set <registry-address> to the registry endpoint of the standby.
Stage the image of the target version and of every intermediate Kubernetes version for each provider in use. The commands that the tabs generate run on global; in DR, run them on both global clusters, on the standby after export KUBECONFIG=<standby-kubeconfig>, and run unset KUBECONFIG afterwards:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-server
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
Upload the qcow2 image of each version to every DCS site that your clusters use, and register it as a VM template with the same name on every site. Then generate the VM template ConfigMaps and apply them on global. Select every architecture whose VM templates you register; each generated ConfigMap records its architecture. Register the VM templates of every architecture that your clusters use, and in a cluster that mixes architectures, set architecture on every ConfigMap, as described in VM Template ConfigMap.
Run the command on the global cluster. In a DR environment, run it on both global clusters: the ConfigMaps are not synchronized between them.
For each ACP version, register VM templates of only one OS per architecture on a global cluster. When VM templates of different OSes have the same Kubernetes version and architecture, an upgrade can move a cluster to the other OS.
ConfigMaps created from earlier versions of this document have no architecture key and no architecture in their name. Once the ConfigMaps of an ACP version exist for each architecture that you use, you can delete the earlier ConfigMaps of that version when no upgrade is in progress.
If you register VM templates of both architectures for a version on the upgrade path, also register the VM template that each cluster runs now, including global, when it has no ConfigMap yet. The upgrade takes the architecture of a node group from the ConfigMap of its current VM template. Without that ConfigMap, the upgrade cannot choose between the two architectures and stops with ImageCatalogMiss. Register it as described in VM Template ConfigMap, with the ACP version of the Alauda OS image that the VM template was made from.
On the execution host, from the installer/ directory of the target Core Package, push the two images of the archive of each version. The host needs free space in /var/tmp of at least the archive size; <host-arch> is its architecture, amd64 or arm64. For Alauda OS M55, push both baremetal-base-image and baremetal-base-image-iso. For Alauda OS KU, push both kubeos-rootfs and kubeos-iso. <os-image-tag> is the archive file name without .tar. The admin password is in the cpaas-system/registry-admin Secret of global. In DR, push to the registry of both global clusters. For a workload cluster with its own registry, push the images into that registry as well, with its address and account; otherwise, the upgrade of that cluster stops with ImageNotFound. See Bind a Workload Cluster to Its Own Registry.
read -rsp "Registry password: " REGISTRY_PASSWORD; printf '\n'# Alauda OS M55for repo in baremetal-base-image baremetal-base-image-iso; do res/<host-arch>/packtool image push <bare-metal-os-archive> \ "<registry-address>/tkestack/${repo}:<os-image-tag>" --ref "${repo}:<os-image-tag>" \ -u "<registry-username>" -p "${REGISTRY_PASSWORD}"done# Alauda OS KUfor repo in kubeos-rootfs kubeos-iso; do res/<host-arch>/packtool image push <bare-metal-os-archive> \ "<registry-address>/tkestack/${repo}:<os-image-tag>" --ref "${repo}:<os-image-tag>" \ -u "<registry-username>" -p "${REGISTRY_PASSWORD}"doneunset REGISTRY_PASSWORD
Before you request the upgrade, add the keys of every version to the image catalog on global, for each OS and architecture of the hosts in the pools. A host replaced while its key is missing stops the upgrade with ImageCatalogMiss until you add the key:
An OS image tag is the file name of its OS archive without .tar, and the tag of both images of the archive.
ac -n cpaas-system patch configmap elemental-image-catalog --type merge --patch-file /dev/stdin <<'EOF'
{
"data": {
"v1.36.4-acp.3": "tkestack/baremetal-base-image:<os-image-tag-for-v4.5.0>",
"v1.36.4-acp.3_etcd": "acp/k8s/etcd:v3.6.13-acp.5"
}
}
EOF
ac -n cpaas-system get configmap elemental-image-catalog -o yaml
The values have no registry address. When the provider installs the OS image on a host, it uses the registry of the cluster, or its registry mirror.
Run the command on the global cluster. In a DR environment, run it on both global clusters: the image catalog is not synchronized between them.
The command adds keys and keeps the existing ones. Do not remove existing keys, including those of the Kubernetes version that a cluster runs today: they are used to add nodes and to replace failed nodes.
Do not change the image catalog while an upgrade is in progress. A host replaced after the change installs the new image, so the nodes of one pool can end up on different images.
Use one Alauda OS variant per cluster. Alauda OS M55 hosts use amd64. The hosts of an Alauda OS KU cluster can mix amd64 and arm64; add the keys of every architecture that its hosts use.
The command creates a ResourcePatch on global. Do not delete it. Exempt it for the target version before the global upgrade, and again for each later minor upgrade, as described in Resolve Preflight Blocks.
List the architecture and OS labels of the hosts with ac -n cpaas-system get machineinventories.elemental.cattle.io -L os.baremetal.alauda.io/arch,os.baremetal.alauda.io/id. The key of a host combines the Kubernetes version with these labels. A host without the labels resolves to amd64 and Alauda OS M55. That needs no action for amd64 hosts with Alauda OS M55; for any other host without the labels, contact technical support.
Upload the qcow2 image of each version to the HCS image service, as an image with the same name in each region and project that your clusters use. The image must meet the image requirements in Alauda OS Image. Then generate the image ConfigMaps and apply them on global, as described in Image ConfigMap. Every Huawei Cloud Stack cluster needs them, including a cluster created with YAML. Select every architecture whose images you register; each generated ConfigMap records its architecture. Register the images of every architecture that your clusters use, and in a cluster that mixes architectures, set architecture on every ConfigMap, as described in Image ConfigMap.
Run the command on the global cluster. In a DR environment, run it on both global clusters: the ConfigMaps are not synchronized between them.
For each ACP version, register images of only one OS per architecture on a global cluster. When images of different OSes have the same Kubernetes version and architecture, an upgrade can move a cluster to the other OS.
ConfigMaps created from earlier versions of this document have no architecture key and no architecture in their name. Once the ConfigMaps of an ACP version exist for each architecture that you use, you can delete the earlier ConfigMaps of that version when no upgrade is in progress.
Keep the ConfigMaps of earlier ACP versions. For each ACP version, register only one image per architecture.
If you register images of both architectures for a version on the upgrade path, also register the image that each cluster runs now, including global, when it has no ConfigMap yet. The upgrade takes the architecture of a node group from the ConfigMap of its current image. Without that ConfigMap, the upgrade cannot choose between the two architectures and stops with ImageCatalogMiss. Register it as described in Image ConfigMap, with the ACP version of the Alauda OS image that it was made from.
In each vCenter datacenter that your clusters use, deploy the ova of each version, and convert the deployed virtual machine to a VM template. Use the same template name in every datacenter. The template must meet the VM template requirements. Then generate the VM template ConfigMaps and apply them on global, as described in VM Template ConfigMap. Every VMware vSphere cluster needs them.
Run the command on the global cluster. In a DR environment, run it on both global clusters: the ConfigMaps are not synchronized between them.
For each ACP version, register VM templates of only one OS on a global cluster. When VM templates of different OSes have the same Kubernetes version, an upgrade can move a cluster to the other OS.
Keep the ConfigMaps of earlier ACP versions. For each ACP version, register only one VM template.
Applies to: every global cluster on Alauda OS, in the tab of its infrastructure provider, as in Check by OS and Provider. In DR, run each block on the standby too, as the comments on its first and last lines show.
Non-Alauda OS
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
No action is needed.
Run this check only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
On global:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get configmap -l cpaas.io/dcs-vm-template -L cpaas.io/dcs-vm-template,cpaas.io/distribution-version# on the standby: unset KUBECONFIG
Continue when, for the target version and for each intermediate Kubernetes version, a ConfigMap shows its ACP version in DISTRIBUTION-VERSION, and, in the DCS console of every site that global uses, a VM template has the name in its DCS-VM-TEMPLATE column; otherwise, run the Huawei DCS steps of Stage the Alauda OS Images for that version.
Run this check only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
On global:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io \ -o custom-columns='NAME:.metadata.name,CLUSTER:.spec.clusterName,HOSTS:.spec.machineInventories[*].name'ac -n cpaas-system get machineinventories.elemental.cattle.io \ -o custom-columns='NAME:.metadata.name,ARCH:.metadata.labels.os\.baremetal\.alauda\.io/arch,OS:.metadata.labels.os\.baremetal\.alauda\.io/id'ac -n cpaas-system get configmap elemental-image-catalog -o yaml# on the standby: unset KUBECONFIG
The hosts of global are the ones in HOSTS of the pools whose CLUSTER is global. For a host whose ARCH or OS is empty or <none>, use amd64 or alaudaos.m.55. Continue when, for each Kubernetes version on the path, the catalog has the key <kubernetes-version>_etcd and, for the ARCH and OS of each host of global, the OS image key <kubernetes-version>_<arch>_<os-id>, or the bare <kubernetes-version> for amd64 with alaudaos.m.55; otherwise, run the Bare Metal steps of Stage the Alauda OS Images for the missing keys.
Run this check only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
On global:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get configmap -l cpaas.io/hcs-vm-template -L cpaas.io/hcs-vm-template,cpaas.io/distribution-version# on the standby: unset KUBECONFIG
Continue when, for the target version and for each intermediate Kubernetes version, a ConfigMap shows its ACP version in DISTRIBUTION-VERSION, and, in the HCS console of every region and project that global uses, exactly one image has the name in its HCS-VM-TEMPLATE column; otherwise, run the Huawei Cloud Stack steps of Stage the Alauda OS Images for that version.
Run this check only when InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared in Run Preflight Checks. Otherwise, InfraClusterVersionPreflight covers these items.
When InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
On global:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get configmap -l cpaas.io/vsphere-vm-template -L cpaas.io/vsphere-vm-template,cpaas.io/distribution-version# on the standby: unset KUBECONFIG
Continue when, for the target version and for each intermediate Kubernetes version, a ConfigMap shows its ACP version in DISTRIBUTION-VERSION, and, in vCenter, every datacenter that global uses has exactly one VM template with the name in its VSPHERE-VM-TEMPLATE column; otherwise, run the VMware vSphere steps of Stage the Alauda OS Images for that version.
Applies to: every global cluster. In DR, run it on each global cluster. Skip it for workload clusters.
From the installer/ directory of the target Core Package, on each global cluster:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverunset REGISTRY_USERNAME REGISTRY_PASSWORDac get productmanifestsbash upgrade.sh --preflight# on the standby: unset KUBECONFIG
Result
Action
Summary shows Passed or PassedWithWarnings, and no InfraClusterVersionPreflight or InfraClusterVersionPreflight/<check> row reports a warning
Continue.
ac get productmanifests reports the server doesn't have a resource type
Continue without bash upgrade.sh --preflight. You run it in Run the Final Preflight Check, after you deploy the Cluster Version Operator (CVO).
On the standby, DRSplitUpgradeWindow is the only failed check, and its message names only EtcdSyncNotPaused, TokenRotatorEnabled, or both
Continue. Summary shows Blocked at this stage.
On the standby, DRSplitUpgradeWindow names another reason
On a global cluster on Alauda OS, InfraClusterVersionPreflight reports ProviderCapabilityNotDeclared
Run the script whose name ends with -no-preflight.sh in the provider tab of Check by OS and Provider, and the check in the provider tab of Check the Staged Images, and handle their results. Then continue.
For ResourcePatchUpgradeable, review each listed patch with technical support, then exempt it on the global cluster, with the target version as the value; do not use *. In DR, do it on each side:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac get resourcepatches \ -o custom-columns='NAME:.metadata.name,EXEMPT:.metadata.annotations.config\.cpaas\.io/exempt-for-ver'ac get resourcepatches <rp-name> -o yamlac annotate resourcepatches <rp-name> config.cpaas.io/exempt-for-ver=<target-version> --overwrite# on the standby: unset KUBECONFIG
For AdminAckRequired, copy the key from admin-gates, record the acknowledgement, and run preflight again. On non-Alauda OS, first complete the node check in the Non-Alauda OS tab of Check by OS and Provider. In DR, do it on each side:
ac config use-cluster global # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>ac whoami --show-serverac -n cpaas-system get configmap admin-gates -o yamlACK_KEY='<key-from-admin-gates>'ac -n cpaas-system patch configmap admin-acks --type merge -p "{\"data\":{\"${ACK_KEY}\":\"true\"}}"bash upgrade.sh --preflight# on the standby: unset KUBECONFIG