Upgrade the Standby global Cluster After the Workload Clusters
Use this page for a patch upgrade within the same minor version, from ACP 4.5.0 or later, when the standby is not upgraded together with the primary: you upgrade the primary and the workload clusters that you choose first, and the standby afterwards. The plugin stays installed and synchronization is paused, so the standby remains a standby. There is no DR protection from the pause until synchronization is resumed and checked, through the upgrades of the primary, of those workload clusters, and of the standby; in that time, make no changes on either global cluster other than those upgrades and the steps on this page. For a minor upgrade, or to upgrade both global clusters before the workload clusters, use Upgrade global Clusters in a DR Environment; Upgrade Order and Compatibility compares the two procedures.
TOC
Before You BeginUpgrade the Primary and the Workload ClustersUpgrade the Standby and Resume SynchronizationNext StepBefore You Begin
- Both
globalclusters run the same ACP version (Distribution Version), 4.5.0 or later, and the target is a later patch version of the same minor version.<target-version>on this page is that target ACP version, for example4.5.1. - Prepare for the Upgrade is complete on both
globalclusters. These items are not synchronized between them, so each side has its own: published packages, including the Global Cluster Disaster Recovery package on the standby; Alauda OS image ConfigMaps or catalog entries; the administrator acknowledgement;ResourcePatchexemptions; and a preflight result that lets you continue. On the standby, every check passes exceptDRSplitUpgradeWindow, which passes after you disable the token rotator and pause synchronization below. - Each
globalcluster has a recent successful etcd backup, as described in Check the etcd Backup. - You are on the execution host, logged in with
ac loginas a platform administrator, in theinstaller/directory of the extracted target Core Package, with a kubeconfig of the standby that points to its VIP:<standby-kubeconfig>.
On the primary, ac commands use the ac login session, and upgrade.sh uses kubectl with the kubeconfig that ac login saved, after ac config use-cluster global; ac whoami --show-server prints https://<platform-address>/kubernetes/global/. Each command block for the standby starts with export KUBECONFIG=<standby-kubeconfig> and ends with unset KUBECONFIG, and ac whoami --show-server prints the standby VIP. Confirm the printed address before you continue with a block.
Upgrade the Primary and the Workload Clusters
Verify Data Consistency
Do not pause synchronization until this check passes.
Run the check on a control plane node of the standby global cluster, then return to the execution host:
Then check the machines and ongoing upgrades. On the primary:
On the standby:
Prepare Both global Clusters
On the standby, restart the cluster-transformer Pods, so that no work started before the pause keeps running on the standby, and disable the token rotator, which can otherwise revoke workload cluster tokens that the primary still uses. The set env command prints The operation is allowed, and ResourcePatch <name> is created; the block exempts that ResourcePatch for the target version, so that preflight does not block on it:
On the primary, disable the token rotator and exempt its ResourcePatch as well: the paused standby does not receive the tokens it writes, and the standby's copies stop working when it deletes the replaced tokens:
Continue when, on both sides, the rollout completes, annotate reports the ResourcePatch as annotated, and the last command prints false, and on the standby every cluster-transformer Pod is Running. Then run the preflight check on the standby:
Pause Synchronization
From the pause until synchronization is resumed and checked, there is no DR protection. Do not fail over to the standby. Do not uninstall the plugin: uninstalling it removes the DR marker kube-public/etcd-sync, which the plugin creates, and the standby then acts on workload clusters. Do not delete or edit the marker or cpaas-system/etcd-sync-control except in Resume Synchronization. On either global cluster, make no changes other than the upgrades and steps on this page: the primary, the workload clusters, and the standby.
On the standby, pause synchronization, with a change reference of your choice as reason, then read the pause state, its message and self-check (status.pauseCheck), and the DR marker:
Run the block again without the lines from if to fi until the second value on the line after the address is Paused. In the table, "the state line" is that line.
Upgrade the Primary global Cluster
An upgrade cannot be rolled back. When the target changes the Kubernetes build or the Alauda OS image, the request also replaces every node with a new VM on Huawei DCS, Huawei Cloud Stack, and VMware vSphere, and reinstalls every host on Bare Metal, one at a time.
On the primary, deploy the Cluster Version Operator (CVO) and the target release information. <target-version> is the target ACP version:
Then run the final preflight check:
Continue when Summary shows Passed or PassedWithWarnings. Otherwise, resolve the checks on the primary as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:
Monitor the upgrade:
Run the commands again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.
Upgrade the Workload Clusters
First, on the primary, check that the token rotator is still disabled after the upgrade:
Continue when it prints false; otherwise, run the primary block of Prepare Both global Clusters again. Then complete Upgrade Validation on the primary.
Upgrade the workload clusters that you choose to upgrade now by following Upgrade Workload Clusters, with these differences, then return to this page:
- Skip Confirm the Standby
globalCluster Is Upgraded. - Skip Check DR Synchronization while synchronization is paused.
- Skip Upgrade the Agnostic Extensions; upgrade them in Upgrade the Agnostic Extensions and Validate, after synchronization is resumed.
Continue when every workload cluster that you upgraded shows the target version in Current with Ready True and has passed its validation, and no workload cluster upgrade is running. Upgrade the other workload clusters after this page is complete.
Upgrade the Standby and Resume Synchronization
Check the State Before Upgrading the Standby
Start here when you return to upgrade the standby. Log in again with ac login if the session expired, and check that each global cluster still has a recent successful etcd backup, as in Before You Begin. Then check that the state is unchanged. On the primary:
Continue when Current shows the target version and Ready is True, on every row of the cvsh list REQUESTED is empty or equal to CURRENT, and the last command prints false. Use this Current value as <target-version> for the standby.
On the standby, check the pause state and the marker, then check from the execution host that the token of the plugin still gives access to the primary. The block reads the primary VIP from the plugin settings and puts an IPv6 address in brackets; the token is passed to curl through standard input and is not printed:
Upgrade the Standby global Cluster
An upgrade cannot be rolled back. When the target changes the Kubernetes build or the Alauda OS image, the request also replaces every node with a new VM on Huawei DCS, Huawei Cloud Stack, and VMware vSphere, and reinstalls every host on Bare Metal, one at a time.
On the standby, deploy CVO and the target release information:
Then run the final preflight check. Check the marker first, because without the marker DRSplitUpgradeWindow passes without checking the standby:
If the first command does not print configmap/etcd-sync, do not upgrade the standby, and contact technical support. Continue when Summary shows Passed or PassedWithWarnings. If DRSplitUpgradeWindow fails, see DR Standby Checks. For other checks, resolve them on the standby as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:
Monitor the upgrade:
Run the block again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.
Resume Synchronization
Resuming makes the standby match the primary: objects that existed at the pause and were deleted on the primary since then are also deleted on the standby. Resume only after the checks below pass.
First, on the standby, check that synchronization is still paused, the marker exists, and the token rotator is still disabled after the upgrade:
Continue when the output is Paused Paused Paused, configmap/etcd-sync, and false. If the token rotator value is not false, run the standby block of Prepare Both global Clusters again. If the pause state or the marker differs, do not resume, and contact technical support. Then resume synchronization on the standby and read the result:
Run the block again without the patch command until the line after the address shows Running. Then run the /check command from Verify Data Consistency again on a standby control plane node.
Restore the Token Rotator
On the primary, remove the setting, then check it and list the ResourcePatch objects for its Deployment:
On the standby:
Continue when, on both sides, the rollout completes, the jsonpath command for the setting prints an empty line, and the last command lists no ResourcePatch: removing the setting also removes the ResourcePatch that it created. If a ResourcePatch is still listed, it holds another customization of the Deployment; keep it, and check that it does not contain CLUSTER_TOKEN_ROTATOR_ENABLED.
Upgrade the Agnostic Extensions and Validate
The upgrade requests do not upgrade Extensions (Operators and Cluster Plugins) with the Agnostic lifecycle. Upgrade each Agnostic Extension installed on the primary, on the standby, and on the workload clusters that you upgraded, by following its own documentation. For example, see Upgrade Alauda AI and Upgrade Alauda DevOps. Then complete Upgrade Validation on the standby; the primary was validated before the workload clusters.
The upgrade of the two global clusters is complete only when every step on this page is done, including the synchronization check after the resume; do not start another upgrade of either global cluster before that.
Next Step
Upgrade the remaining workload clusters. See Upgrade Workload Clusters.