Create a Global Cluster Disaster Recovery Pair
This page describes how to create a disaster recovery (DR) pair of global clusters, from planning to the first synchronization. Read it before you write the first install-config.yaml for either side.
TOC
What a DR Pair IsDecide Before You InstallPrepare for DRGenerate the Shared KeysNetworkObservability PlacementWrite the Two install-config.yaml FilesInstall the Primary and the StandbyStart SynchronizationConventions1. Install etcd-sync2. Start and Verify SynchronizationBare Metal Only: Check the Shared Agent Identity3. Install the Cluster PluginsKeep It HealthyBare Metal: Agents and the Platform AddressNextWhat a DR Pair Is
A DR pair consists of two global clusters: a primary global cluster and a standby global cluster. The primary global cluster serves the platform access address, which is a domain name. In normal operation, the domain points to the primary global cluster. The exceptions are while you install the standby global cluster and during a failover. The other cluster is the standby global cluster. After a failover, the former standby global cluster becomes the new primary global cluster.
You install each side as an ordinary installation with ACP CLI (ac), from its own bootstrap node and with its own install-config.yaml. After both sides are installed, Global Cluster Disaster Recovery (etcd-sync) runs on the standby global cluster and copies the etcd data of the primary global cluster to it. ac prepares every global cluster for DR. You do not need to edit any manifests for a DR pair. The infrastructure credentials of each side are not synchronized. The two sides can therefore use different infrastructure accounts.
This page applies to every provider. Steps that only Bare Metal needs are marked Bare Metal only.
Decide Before You Install
Decide on DR before you write the first install-config.yaml for either side. The standby global cluster can decrypt the Secrets of the primary global cluster only with the encryption key of the primary global cluster. An installed cluster keeps the key that it was installed with.
- If a
globalcluster is already installed as a single cluster, contact technical support before you install a standbyglobalcluster. - Do not install a standby
globalcluster with its own key. Such a standbyglobalcluster cannot decrypt the Secrets of the primaryglobalcluster.
Prepare for DR
Each side has its own:
- Bootstrap node.
- Infrastructure.
- Control plane VIP.
Both sides must use the same global.registry address and account. Use one of these setups:
- One external registry that both sides share.
- One registry per side, behind one domain name.
Workload clusters that use the global registry keep this address and account after a failover. If the values differ between the two sides, those workload clusters cannot use the registry of the new primary global cluster.
Each global cluster keeps its own registry configuration, and synchronization does not copy it. When the registry password changes, update it on both global clusters.
Each side meets the common prerequisites and the prerequisites of its provider.
Also prepare:
- The platform domain name. It is the platform access address. During normal operation, it resolves only to the primary VIP, and the standby
globalcluster reaches the platform of the primaryglobalcluster through it. You point it at a VIP three times during the installation and once more at a failover; see Install the Primary and the Standby. Schedule these changes in advance. - A user-provided certificate and its private key for the domain, with the same files on both bootstrap nodes.
- The registry domain name, if you use one registry per side. It is the value of
global.registry.address. It resolves to the registry of the side that the platform domain points at. You switch both domains together. - The shared keys, generated once; see Generate the Shared Keys.
- A copy of each side's installation directory, in particular
output/kubeconfig. Store it where you can reach it when a site is down. - Violet v4.2.13 or later, on a machine that can reach the standby VIP and the standby registry. The
etcd-syncpackage is a component of Global Cluster Disaster Recovery and is shipped in ACP Common Aligned Extensions; see Install etcd-sync.
Generate the Shared Keys
Generate each key once and write the same value in both install-config.yaml files. A key file holds a secret. Run chmod 600 on it and keep it private.
Run the commands on a Linux machine with OpenSSL. To generate the encryption key for Secrets at rest:
Add the key to both install-config.yaml files. Write the key itself on one line, not the file path. To print it, run cat dr-encryption.key.
ac uses the value as it is, and checks it before it changes anything. The value must be the base64 encoding of a 16, 24, or 32 byte key, which the command above produces.
Bare Metal only: to generate the key that signs ServiceAccount tokens:
You cannot change the signing key after installation. If the cluster already holds a different global-sa, ac stops instead of overwriting it.
Network
Before you install either cluster, ask your network administrator to allow the flows in Global Disaster Recovery, in both directions between the networks of the two sides.
The nodes of each side pull their images from the registry in global.registry. For the flows to that registry, see Image Registry.
Observability Placement
DR replicates only the etcd data of the global cluster. Metrics and logs are stored outside etcd, so they are never copied to the standby global cluster. Plan observability before you install either cluster:
- Each
globalcluster collects, stores, and queries its own metrics and logs. The standbyglobalcluster can therefore observe itself immediately after a failover. - Do not send the monitoring data and logs of workload clusters to either
globalcluster. Aggregate them in a dedicated workload cluster that runs the VictoriaMetrics Center and the log storage. If a network zone cannot reach that cluster, give the zone its own aggregation cluster.
For what is synchronized and what is excluded, see Global Cluster Disaster Recovery.
Write the Two install-config.yaml Files
Write the install-config.yaml of each side as for a single global cluster. Follow the Create the global Cluster page of your provider: Bare Metal, Huawei DCS, VMware vSphere, or Huawei Cloud Stack. Set these values:
- Platform address: use a domain name, not an IP address.
acaccepts an IP address here, butetcd-syncrefuses to run when the platform address is not a domain name. - Certificate: do not omit
global.platform.cert. DR does not support self-signed certificates. Without the field, each side generates its own self-signed certificate. After a failover, clients then reject the certificate of the other side. On Bare Metal, these clients include the agents of the workload hosts. For the accepted formats, see Commoninstall-config.yamlFields. extraAddresses: each side lists its own VIP. You use that VIP to reach the side while the domain points at the other side.acdoes not add the VIP.- Keys: write them before the first
manifestsorcreaterun on each side. If you change a key afterward, an installed cluster keeps the key that it was installed with; see Rendered Manifests. - VM folder (VMware vSphere): when both sides share a vCenter, you can set a different
provider.vsphere.folderon each side to keep their VMs apart. The same folder also works.
Applies to: every DR pair, before the first manifests or create run.
Run this command on each bootstrap node, in the installation directory:
On the machine where you keep the shared keys, also run the following command. It drops the trailing newline of the file and prints the hash of the key. Compare the hashes of both sides with this one:
Continue when both sides print the same hash as the key file. On a side that prints nothing or another hash, set cluster.encryptionProviderConfig in its install-config.yaml to the content of dr-encryption.key.
Bare Metal only. Also run this command on each bootstrap node, in the installation directory:
On the machine where you keep the shared keys, also run the following command:
Continue when both sides print the same hash as the key file. On a side that prints nothing or another hash, set provider.baremetal.serviceAccountSigningKey in its install-config.yaml to the content of sa.key.b64.
Install the Primary and the Standby
While you install a side, the platform domain must point to that side. To reach the other side meanwhile, use its VIP and the kubeconfig in its installation directory.
If you use one registry per side, point the registry domain at the same side as the platform domain in each step.
When both sides are installed and the domain points at the primary global cluster, continue with Start Synchronization.
Start Synchronization
After both sides are installed, install Global Cluster Disaster Recovery (etcd-sync) on the standby global cluster and verify the synchronization. The plugin copies the etcd data of the primary global cluster to the standby global cluster.
Before you start:
- The platform domain points at the primary VIP.
- Both sides were installed with the same
cluster.encryptionProviderConfig; see Generate the Shared Keys. If they were not, contact technical support before you continue. - Applies to workload clusters on infrastructure other than that of the
globalcluster. Install the provider of that infrastructure on bothglobalclusters, so that the standbyglobalcluster can reconcile the same workload clusters after a failover. See Prepare. Upload the provider to the standbyglobalcluster with--platform-addressand--dest-reposet to the standbyglobalcluster, as in step 1. If no workload cluster runs on other infrastructure, skip this item.achas already installed the provider of its own platform on each side.
Conventions
The following steps run commands against both global clusters from one machine. Use a machine that has ac and can reach the VIPs of both sides, and copy the kubeconfig of each side to it. The kubeconfig of a side is output/kubeconfig in its installation directory and connects to the VIP of that side.
- This page calls the two files
<primary-kubeconfig>and<standby-kubeconfig>. Each command block starts withexport KUBECONFIG=<primary-kubeconfig>orexport KUBECONFIG=<standby-kubeconfig>and ends withunset KUBECONFIG. - To show which cluster a command talks to, run
ac whoami --show-server. - In the parameter names of the plugin, Active means the primary
globalcluster.
The step marked Bare Metal only is not numbered.
1. Install etcd-sync
Install Global Cluster Disaster Recovery (etcd-sync) only on the standby global cluster, after both sides have completed create. Never install it on both sides. Use the version released with 4.5.
Reach the standby global cluster through its VIP.
-
Get the plugin package from ACP Common Aligned Extensions and upload it to the standby
globalcluster with Violet v4.2.13 or later.- Violet signs in to the standby
globalcluster with an API token. Create the token in the Web Console of the standbyglobalcluster (https://<standby-vip>): Profile → API Tokens. - Set both
--platform-addressand--dest-repoto the standbyglobalcluster. Otherwise, the package goes to the primaryglobalcluster. --dest-repois the address of the registry that the standbyglobalcluster uses. With one shared registry, it isglobal.registry.address. With one registry per side, the registry domain now resolves to the primary registry, so use an address that reaches the standby registry instead.- If the standby registry needs no credentials, use
--no-authinstead of the registry credentials.
For the flags, see Upload Packages.
- Violet signs in to the standby
-
Create the token Secret on the standby
globalcluster. The plugin uses this token to read the primaryglobalcluster; you do not log in with it. Its value is the bearer token of thecpaas-system/k8sadminServiceAccount on the primaryglobalcluster. The commands do not print the token: -
Open the Web Console of the standby
globalcluster athttps://<standby-vip>, switch to the Administrator view, go to Marketplace → Cluster Plugins, select theglobalcluster, and install Global Cluster Disaster Recovery with these parameters.
2. Start and Verify Synchronization
First check that the plugin is running on the standby global cluster:
Continue when the ModuleInfo shows STATUS Running and every Pod is Running. Otherwise, wait and run the commands again. If the results do not change, run the steps in Global Cluster Disaster Recovery Does Not Synchronize.
The mirror copies the data of the primary global cluster after its start delay. Wait until the initial synchronization has finished:
Continue when the output contains Start Sync update. Otherwise, wait and run the commands again.
Then check the synchronization. On your machine, read the address of the monitor:
Run the check on a control plane node of the standby global cluster. Put an IPv6 address in brackets, for example [fd00::10]:
To restart the leader Pod, run the following commands. They read the name of the current leader Pod from the etcd-sync-mirror Lease on the standby global cluster and delete that Pod. Run them only when the table above tells you to:
Bare Metal Only: Check the Shared Agent Identity
Check the shared workload agent identity on each side:
Continue when all four objects exist on both sides, or when none of them exists on either side. Otherwise, contact technical support.
3. Install the Cluster Plugins
Installing a global cluster does not install every cluster plugin, for example Advanced Monitoring. Install the cluster plugins that you need only now, on both global clusters, after both clusters are installed and synchronization runs.
Keep It Healthy
- Run the check in step 2 regularly. The plugin also runs the check by itself, every hour by default, and reports the results as the metrics
local_missed_keys_countandlocal_surplus_keys_count. - After each change to your workload clusters, save the output of
ac get clusters.cluster.x-k8s.io -Aandac get machines.cluster.x-k8s.io -Afrom the primaryglobalcluster. After a failover, you compare the new primaryglobalcluster against this output; see Fail Over. - A DR-aware upgrade of the
globalclusters uninstalls the plugin and has its own consistency check; see UpgradeglobalClusters in a DR Environment.
Bare Metal: Agents and the Platform Address
Bare Metal only. The agent of a global host talks only to the API server of its own global cluster, through the VIP of that cluster. The agent of a workload cluster host talks to the platform access address, not to the VIP of its cluster. After a failover, it therefore reaches the new primary global cluster without any change.
To add or replace a global host after the installation, register it on the global cluster that it belongs to. Follow Hosts for the global Cluster, which describes the naming rule for the registration and the annotations that point the agent of the host at the local API server.