Upgrade Validation

Validate each upgraded cluster to finish its upgrade. In a DR environment, validate the standby and the primary global cluster separately, before you upgrade any workload cluster; when the standby is upgraded after the workload clusters, validate the primary before the workload clusters and the standby after its upgrade. The upgrade of a cluster is finished when every check on this page passes.

Run the commands on this page from the execution host, on the global cluster, which is the primary in a DR environment, for the global cluster and for every workload cluster; ac whoami --show-server prints an address that ends with /kubernetes/global/:

ac config use-cluster global
ac whoami --show-server

To validate the standby global cluster in a DR environment, run export KUBECONFIG=<standby-kubeconfig> instead, confirm that ac whoami --show-server prints the standby VIP, keep it for every block on this page, skip the ac config use-cluster lines, and run unset KUBECONFIG when you finish.

Check the ACP Version

Replace <cluster-name> with global or the workload cluster name:

ac adm upgrade status --cluster=<cluster-name>
ac -n cpaas-system get cvsh <cluster-name> \
  -o jsonpath='{range .status.history[*]}{.version}{"\t"}{.state}{"\t"}{.startedTime}{"\t"}{.completionTime}{"\n"}{end}'

Continue when Upgrade is Completed, Current shows the target ACP version (Distribution Version), the Ready condition is True, Stalled is not True, and the first history entry shows the target version with the state Completed; otherwise, follow Inspect Upgrade State.

Check Kubernetes, Nodes, and Extensions

On the global cluster, read the Core and the installed Extensions of the cluster:

ac get moduleinfo -l cpaas.io/cluster-name=<cluster-name> \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,PHASE:.status.phase'
ac get clustermodule <cluster-name> -o jsonpath='{.status.base.deployStatus}{"\n"}'

Then read Kubernetes, the nodes, and the Pods on the upgraded cluster itself. For a workload cluster, the first line switches to it and the last line switches back to global; for a global cluster, skip both lines:

ac config use-cluster <cluster-name>   # workload cluster only
ac version
ac get nodes
ac get pods --all-namespaces \
  | awk '{if ($4 != "Running" && $4 != "Completed")print}' \
  | awk -F'[/ ]+' '{if ($3 != $4)print}'
ac config use-cluster global           # workload cluster only
ResultAction
Every module is Running with CURRENT equal to TARGET, and deployStatus is not Upgrading, DeployFailed, or UpgradeFailed. Every node is Ready; for a cluster created by ACP, the server version and the VERSION of every node are the Kubernetes version of the target ACP release. The ac get pods command prints only its header lineContinue.
The ac get pods command lists only Pods of your own applications that did not run before the upgrade eitherContinue.
A module is not Running or its CURRENT differs from TARGET, or deployStatus is Upgrading, DeployFailed, or UpgradeFailedFollow Diagnose a Module That Does Not Advance.
deployStatus is empty or <none>Run the command again later. If it stays empty, contact technical support.
Another node or Pod resultContact technical support.

Additional Checks for Alauda OS Clusters

Applies to: Alauda OS clusters. Skip it for non-Alauda OS clusters.

Check the Kube-OVN and CoreDNS Takeover

On the global cluster, use the kind that matches the provider of the cluster, dcscluster, baremetalcluster, hcscluster, or vspherecluster:

ac -n cpaas-system get <provider-cluster-kind> <cluster-name> \
  -o jsonpath='{range .status.conditions[?(@.type=="CoreAddonsHandedOff")]}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
ResultAction
True, or no output on a cluster that does not use Kube-OVNContinue.
False with CoreAddonsHandoffPending or CoreDNSSkipPhasePendingRun the command again later; if it does not change, contact technical support.
False with CoreAddonsHandoffNotApplicableContact technical support.
False with CoreAddonsHandoffBlockedContact technical support with the message, which names the setting.

Check the Provider Resources

On the global cluster, run the checks for the provider of the cluster:

Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
ac -n cpaas-system get infraclusterversion <cluster-name>
ac -n cpaas-system get infraclusterversion <cluster-name> \
  -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{"\t"}{.message}{"\n"}{end}'
ResultAction
CURRENT equals DESIRED, PHASE is Succeeded, and Ready is TrueContinue.
The Ready message starts with kube-proxy rollout pendingRun the commands again after the kube-proxy DaemonSet in kube-system of the cluster has rolled out; if the message does not change, contact technical support.
Another resultFollow Read the Progress.

Check the Web UI

Open the platform Web UI. Continue when login, cluster pages, and Marketplace pages load, and, for each Extension with the Agnostic lifecycle that you upgraded, its pages open; otherwise, contact technical support.

Next Step

After you validate the global clusters, continue with Upgrade Workload Clusters. After you validate a workload cluster, return to that page for the next cluster. After the last cluster, continue with After All Clusters Are Upgraded.

After All Clusters Are Upgraded

When global and every workload cluster run the target ACP version:

  • If the source ACP version was earlier than 4.3 and the PKCE hardening is not done yet, follow Disabling the PKCE Plain Method.
  • Delete a VM template, cloud image, or Bare Metal OS image of an earlier version only after the last cluster that uses it is upgraded.