Multi-Cluster
The global cluster runs only the platform, and business workloads run on workload clusters. The global cluster uses a primary and a standby for disaster recovery. For the assumptions that apply to every scenario, see Common Assumptions.
This page also covers an optional Ceph setup for workload clusters.
TOC
Resource RequirementsMachinesDisksCentral RegistryNode PoolsControl Plane EndpointSolution SelectionFeatures and StoragePlugins on Infra NodesInstallation StepsMulti-Cluster with CephMachines and DisksChanges to the Workload ClusterResource Requirements
Machines
global cluster: the table lists one side. Prepare it for both the primary and the standby, or for one side without disaster recovery.
- For more than 50 managed clusters, select the control plane tier in Evaluating Resources for Global Cluster.
- If Harbor is the central registry, plan its machines as described in Central Registry.
Workload cluster: size A matches the tier of 24 worker nodes and 500 namespaces in Control Plane Node Sizing, and size B matches the tier of 120 worker nodes and 1000 namespaces. For a different number of worker nodes or namespaces, select the matching tier on that page.
Size A, for 24 worker nodes and 500 namespaces:
Size B, for 120 worker nodes and 1000 namespaces, with separate pools for the gateway, monitoring, and logging:
- The tables give the minimum recommended size of each node. Physical servers are usually much larger, so select servers that are at least this size.
- The sizes follow the capacity tiers of each product component. After the installation, scale up according to the actual utilization.
- For ARM nodes, scale the sizes as described in Machines.
- Logging is planned for 5,000 log entries per second with 7 days of retention. For more logs, expand the disks.
- For more requests than the gateway pool handles under light load, scale up its nodes.
- The number and size of worker nodes depend on the business workloads and are not covered here.
- 4 cores and 8 GB is the x86 minimum for the bootstrap node. For its disk and the arm64 minimum, see Bootstrap Node.
- If no image registry is available, use the bootstrap node of each side as the platform image registry; see Using the Bootstrap Node as mirror-registry. With many workload clusters, use a highly available registry instead, because one mirror-registry is a single point of failure.
- Alauda Storage can also serve as the main storage of a large platform. To size such a deployment, contact Alauda.
Disks
Sizes are per node, in GB. For what each disk holds and why it has this size, see Disk Configuration. The bootstrap node needs 300 GB or more for the extracted packages and /var/cpaas.
global cluster:
Workload cluster, size A:
Workload cluster, size B:
- On IaaS, the raw disk for TopoLVM is sized for the current need, because it can be expanded later. See Raw Disk for TopoLVM.
Central Registry
A Multi-Cluster platform uses one central registry for application images, shared by the workload clusters.
- A central registry already exists: use it, and skip the planning and installation of Harbor.
- No central registry exists: use Harbor, in one of the following ways:
- Only the registry is needed: install Harbor on the
globalcluster. For its machines and disks, see Harbor on the global Cluster. - Other DevOps tools are also needed: run Harbor in a dedicated DevOps cluster with the other tools. For its machines and disks, see Dedicated DevOps Cluster.
- Only the registry is needed: install Harbor on the
A Harbor standby is optional with disaster recovery. By default, install Harbor on the primary global cluster only. If a Harbor standby is needed, prepare the same machines on the standby as well.
Node Pools
- Keep
deployModeat the default valuenormalon every cluster. For the fields, see Installation Reference. - For the infra pools of the
globalcluster and of size A workload clusters, declare the label and taint on the node pool after the cluster is created. See Create Infra Nodes on Alauda OS Nodes. - For size B workload clusters, create one node pool each for gateway, monitor, and log, with its own role label and taint. See Custom Role Nodes.
Control Plane Endpoint
The global cluster and every workload cluster need a control plane endpoint. Choose one of the following:
- Self-built VIP: needs no external device.
- Enterprise load balancer: allocate one VIP on it for each control plane endpoint. The VIP of the application gateway can also come from the same load balancer.
- Huawei Cloud Stack: use the ELB that the provider creates.
For what to prepare, see Confirm Install-Time Decisions and Plan the Control Plane Endpoint.
Solution Selection
Features and Storage
- On Huawei Cloud Stack, use the host network and the HCS ELB for the gateway VIP instead of MetalLB, as for a Single Cluster.
- For the registry, see Central Registry.
- Each cluster stores only its own monitoring data and logs; do not send them to the
globalcluster. Disaster recovery does not copy them to the standby; see Observability Placement.
Plugins on Infra Nodes
global cluster, the same on the primary and the standby:
Workload cluster, size A:
Workload cluster, size B:
Installation Steps
- For disaster recovery, step 2 also covers the keys that both sides share and the network flows between the sides.
- On size B workload clusters, use the role label and taint of the gateway, monitor, or log pool in place of the infra label and taint when you place each component.
Multi-Cluster with Ceph
When a workload cluster needs storage that follows the Pod across nodes, deploy a small Ceph cluster inside that workload cluster, on a dedicated storage node pool.
Machines and Disks
- Spread the storage nodes across failure domains.
- On Bare Metal, use 4 storage nodes, because a replacement for a failed server can take a long time to arrive. With 3 nodes, Ceph runs degraded on 2 nodes until then, which blocks maintenance and upgrades. With 4 nodes, 3 remain after one failure, and Ceph can restore all 3 replicas.
Sizes are per node, in GB. For what each disk holds and why it has this size, see Disk Configuration.
- With 3 replicas, usable capacity is about one third of the raw OSD capacity. Add OSD disks for more capacity.
/var/lib/rookmust be on a persistent disk, and each OSD must be a separate raw disk. See Prerequisites for Deploying on Alauda OS.- For the CPU and memory that Ceph reserves, see the small three-node baseline in Aggregate Cluster Planning Budget.
Changes to the Workload Cluster
For the installation, see Install.