Fail Over to the Standby global Cluster
Use this page when the primary global cluster has failed, or when you switch to the standby global cluster on purpose. In step 2, you uninstall Global Cluster Disaster Recovery (etcd-sync, "the plugin") from the standby global cluster. In step 3, the standby global cluster becomes the new primary global cluster. In step 6, the former primary global cluster becomes the new standby global cluster. For the definitions of primary and standby, see What a DR Pair Is.
While the plugin is uninstalled, there is no DR protection. Step 6 restores it. To make the former primary global cluster primary again later, fail over again; see step 6.
Have the following ready:
- A machine that has
acand can reach the standbyglobalcluster. <standby-kubeconfig>:output/kubeconfigin the installation directory of the standbyglobalcluster.- SSH access to a control plane node of the standby
globalcluster. - The platform administrator account.
- Someone who can change the DNS records of the platform domain and, if you use one registry per side, of the registry domain.
- The list of workload clusters and
Machineobjects that you saved from the primaryglobalcluster; see Keep It Healthy.
Through step 3, commands for the standby global cluster start with export KUBECONFIG=<standby-kubeconfig> and end with unset KUBECONFIG. To show which cluster a command talks to, run ac whoami --show-server.
Planned failover only. Before step 1, run this command on both global clusters:
Then, on the primary global cluster, check that no workload cluster upgrade is in progress:
Continue when all of the following are true:
- Every machine on both clusters is
Running. - On every row of the
cvshlist,REQUESTEDis empty or equal toCURRENT, or the command reportsthe server doesn't have a resource type. - The flows in Global Disaster Recovery are allowed in both directions.
If REQUESTED differs from CURRENT on a row, wait until the upgrade of that workload cluster completes, then run the block again. If the upgrade does not advance, see Inspect Upgrade State. If any other condition is not true, contact technical support.
TOC
Procedure1. Isolate the Primary2. Check Consistency and Uninstall the Plugin3. Switch the Platform and Registry Domains4. Log In Again and Restart the Dependent Pods5. Check the New Primary6. Make the Former Primary the New StandbyTroubleshootingProcedure
1. Isolate the Primary
Isolate the primary global cluster so that it cannot change your workload clusters:
- If the primary
globalcluster has failed, confirm that it is powered off, or that its VIP cannot be reached from the workload networks. - For a planned failover, block the VIP of the primary
globalcluster from the workload networks. Do not stop the control plane of the primaryglobalcluster before step 2 is complete, because the check in step 2 needs it.
If you do not control the firewall, ask your network administrator. Continue when the VIP of the primary global cluster cannot be reached from the workload networks, or when the primary global cluster is powered off. Otherwise, repeat the isolation.
2. Check Consistency and Uninstall the Plugin
If the check reports missing or surplus keys, do not uninstall the plugin. Resolve the inconsistency or contact technical support. Workload nodes are backed by Cluster API Machine objects, and an inconsistent synchronization can delete those objects and destroy the machines behind them.
Applies to: a primary global cluster that you can still reach. If the primary global cluster is unreachable, the check cannot run, and this page cannot confirm that the standby global cluster holds all the data of the primary global cluster. In that case, contact technical support before you uninstall the plugin. Give them the time when the primary global cluster stopped, and the time and result of the last successful check.
On your machine, read the address of the monitor:
Run the check on a control plane node of the standby global cluster. Put an IPv6 address in brackets, for example [fd00::10]:
To read the log of the leader and, if missed keys remain, restart the leader:
When the check passes, uninstall the plugin from the standby global cluster:
Continue when the last two commands report No resources found. Otherwise, run them again after one minute. If they still list objects, contact technical support.
Scaling the Deployments to zero is not enough. The ModuleInfo still exists, and at step 6 the new standby global cluster refuses to run with this error:
Run the ac delete moduleinfo command above on the new primary global cluster, which is the cluster you are switching to. The new standby global cluster then starts normally.
Check that the marker of the plugin is gone:
Continue when the command reports NotFound. Otherwise, wait and run it again. The providers do not reconcile workload clusters while the marker exists. They resume about one minute after the marker is gone. On Bare Metal, the provider also takes over the shared agent identity at that time. Until it does, new workload hosts cannot obtain their plans.
While the plugin is uninstalled, there is no DR protection. Until step 6 is complete, do not create, scale, or delete clusters or nodes on either global cluster, and do not change cluster configuration there. If the former primary global cluster cannot be recovered, contact technical support before you change any cluster.
3. Switch the Platform and Registry Domains
Switch the platform domain to the standby VIP. If you use one registry per side, also switch the registry domain to the registry of the standby global cluster. The standby global cluster is now the new primary global cluster. Check that the platform on the new primary global cluster resolves the domains to the new addresses:
Continue when the platform domain resolves to the standby VIP and the registry domain resolves to the registry of the standby global cluster. Otherwise, correct the DNS record and run the check again.
4. Log In Again and Restart the Dependent Pods
The platform domain now reaches the new primary global cluster. Log in again instead of reusing a session from before the switch. Then restart the following Pods, if they are installed, so that they reload the DR and endpoint configuration:
If workload clusters send monitoring data to the global cluster, also restart warlock in each workload cluster:
5. Check the New Primary
On the new primary global cluster, verify that every workload Cluster and Machine object is intact. If a Machine object is lost, its node is deleted.
Continue when every workload cluster and Machine in the list that you saved before the failover is listed, and the provider checks below are met. Otherwise, contact technical support. To list the objects of a provider, run ac get <kind> -A.
6. Make the Former Primary the New Standby
Once the former primary global cluster is healthy, make it the new standby global cluster to protect the new primary global cluster again. Keep it isolated from the workload networks until the plugin runs on it. Until the marker of the plugin exists, the providers of a global cluster reconcile workload clusters.
If the former primary global cluster has to be installed again, contact technical support first.
Follow Start Synchronization with these values:
Bare Metal only. A failover changes no plan, so step 5 cannot show that the new primary global cluster delivers new plans. To check it after step 6, register a new host with a workload ISO whose URL uses the platform domain, as described in Register Hosts, and add one worker node to a workload cluster on it, as described in Manage Nodes on Bare Metal. The check passes when the new Machine reaches Running. Otherwise, contact technical support.
After step 6, the two sides are synchronized. To make the former primary global cluster primary again once it is healthy, fail over again. Follow this page again with the roles exchanged.