Multi-Cluster

The global cluster runs only the platform, and business workloads run on workload clusters. The global cluster uses a primary and a standby for disaster recovery. For the assumptions that apply to every scenario, see Common Assumptions.

This page also covers an optional Ceph setup for workload clusters.

Resource Requirements

Machines

global cluster: the table lists one side. Prepare it for both the primary and the standby, or for one side without disaster recovery.

Node poolNodes per sideCPUMemoryPurpose
bootstrap14 cores8 GBRuns the installer, and can also serve as the platform image registry.
control-plane316 cores32 GiBControl plane and platform components, for up to 50 managed clusters.
infra312 cores24 GiBMonitoring and logging of the global cluster itself.

Workload cluster: size A matches the tier of 24 worker nodes and 500 namespaces in Control Plane Node Sizing, and size B matches the tier of 120 worker nodes and 1000 namespaces. For a different number of worker nodes or namespaces, select the matching tier on that page.

Size A, for 24 worker nodes and 500 namespaces:

Node poolNodesCPUMemoryPurpose
control-plane34 cores16 GiBControl plane and platform components.
infra316 cores48 GiBLogging, monitoring, and the application gateway.

Size B, for 120 worker nodes and 1000 namespaces, with separate pools for the gateway, monitoring, and logging:

Node poolNodesCPUMemoryPurpose
control-plane38 cores32 GiBControl plane and platform components.
gateway24 cores8 GiBApplication gateway (Envoy Gateway), planned for light load.
monitor38 cores16 GiBMonitoring.
log38 cores16 GiBLogging.
  • The tables give the minimum recommended size of each node. Physical servers are usually much larger, so select servers that are at least this size.
  • The sizes follow the capacity tiers of each product component. After the installation, scale up according to the actual utilization.
  • For ARM nodes, scale the sizes as described in Machines.
  • Logging is planned for 5,000 log entries per second with 7 days of retention. For more logs, expand the disks.
  • For more requests than the gateway pool handles under light load, scale up its nodes.
  • The number and size of worker nodes depend on the business workloads and are not covered here.
  • 4 cores and 8 GB is the x86 minimum for the bootstrap node. For its disk and the arm64 minimum, see Bootstrap Node.
  • If no image registry is available, use the bootstrap node of each side as the platform image registry; see Using the Bootstrap Node as mirror-registry. With many workload clusters, use a highly available registry instead, because one mirror-registry is a single point of failure.
  • Alauda Storage can also serve as the main storage of a large platform. To size such a deployment, contact Alauda.

Disks

Sizes are per node, in GB. For what each disk holds and why it has this size, see Disk Configuration. The bootstrap node needs 300 GB or more for the extracted packages and /var/cpaas.

global cluster:

Diskcontrol-planeinfra
IaaS
System disk100100
/var/lib/etcd20—
/var/lib/containerd150150
/var/lib/kubelet200200
/var/cpaas (persistent)100200
Raw disk for TopoLVM—650
Bare Metal
System disk, minimum / recommended512 / 1 TB512 / 1 TB
Raw disk for TopoLVM—1 TB

Workload cluster, size A:

Diskcontrol-planeinfra
IaaS
System disk100100
/var/lib/etcd20—
/var/lib/containerd150150
/var/lib/kubelet100100
/var/cpaas (persistent)100200
Raw disk for TopoLVM—650
Bare Metal
System disk, minimum / recommended512 / 512512 / 1 TB
Raw disk for TopoLVM—1 TB

Workload cluster, size B:

Diskcontrol-planegatewaymonitorlog
IaaS
System disk100100100100
/var/lib/etcd20———
/var/lib/containerd150150150150
/var/lib/kubelet100100100100
/var/cpaas (persistent)1005050200
Raw disk for TopoLVM——100600
Bare Metal
System disk, minimum / recommended512 / 512512 / 512512 / 512512 / 1 TB
Raw disk for TopoLVM——5121 TB
  • On IaaS, the raw disk for TopoLVM is sized for the current need, because it can be expanded later. See Raw Disk for TopoLVM.

Central Registry

A Multi-Cluster platform uses one central registry for application images, shared by the workload clusters.

  • A central registry already exists: use it, and skip the planning and installation of Harbor.
  • No central registry exists: use Harbor, in one of the following ways:
    • Only the registry is needed: install Harbor on the global cluster. For its machines and disks, see Harbor on the global Cluster.
    • Other DevOps tools are also needed: run Harbor in a dedicated DevOps cluster with the other tools. For its machines and disks, see Dedicated DevOps Cluster.

A Harbor standby is optional with disaster recovery. By default, install Harbor on the primary global cluster only. If a Harbor standby is needed, prepare the same machines on the standby as well.

Node Pools

  • Keep deployMode at the default value normal on every cluster. For the fields, see Installation Reference.
  • For the infra pools of the global cluster and of size A workload clusters, declare the label and taint on the node pool after the cluster is created. See Create Infra Nodes on Alauda OS Nodes.
  • For size B workload clusters, create one node pool each for gateway, monitor, and log, with its own role label and taint. See Custom Role Nodes.

Control Plane Endpoint

The global cluster and every workload cluster need a control plane endpoint. Choose one of the following:

  • Self-built VIP: needs no external device.
  • Enterprise load balancer: allocate one VIP on it for each control plane endpoint. The VIP of the application gateway can also come from the same load balancer.
  • Huawei Cloud Stack: use the ELB that the provider creates.

For what to prepare, see Confirm Install-Time Decisions and Plan the Control Plane Endpoint.

Solution Selection

Features and Storage

FeatureSelectionStorage
MonitoringVictoriaMetrics, one per clusterTopoLVM
LoggingClickHouse, one per clusterTopoLVM
Application gateway (workload clusters)Envoy Gateway (Gateway API)—
Application gateway VIP (workload clusters)An enterprise load balancer if available; otherwise MetalLB—
Registry for application imagesA central registry: the existing one, or HarborCeph, for Harbor
  • On Huawei Cloud Stack, use the host network and the HCS ELB for the gateway VIP instead of MetalLB, as for a Single Cluster.
  • For the registry, see Central Registry.
  • Each cluster stores only its own monitoring data and logs; do not send them to the global cluster. Disaster recovery does not copy them to the standby; see Observability Placement.

Plugins on Infra Nodes

global cluster, the same on the primary and the standby:

Node poolPlugins
infraAlauda Container Platform Advanced Monitoring, Alauda Container Platform Log Storage Management, Alauda Data Services Analytical Database E1, Alauda Data Services Streaming E1

Workload cluster, size A:

Node poolPlugins
infraAlauda Container Platform Advanced Monitoring, Alauda Container Platform Log Storage Management, Alauda Data Services Analytical Database E1, Alauda Data Services Streaming E1, Alauda Container Platform Gateway API Controller

Workload cluster, size B:

Node poolPlugins
gatewayAlauda Container Platform Gateway API Controller
monitorAlauda Container Platform Advanced Monitoring
logAlauda Container Platform Log Storage Management, Alauda Data Services Analytical Database E1, Alauda Data Services Streaming E1

Installation Steps

StepTaskDocumentation
1Plan, and confirm which network flows apply.Plan, Communication Matrix
2Decide on disaster recovery before you write the first install-config.yaml.Create a Global Cluster Disaster Recovery Pair, Decide Before You Install
3Download the installation packages.Download
4Prepare the platform image registry, and upload the images to both sides for disaster recovery.Prepare an External Platform Image Registry, mirror-registry
5Meet the common and provider prerequisites, with one bootstrap node on each side.Prerequisites, Provider Pages
6Plan the disks of each node pool before you create the clusters.Fixed at Creation, Disk Configuration
7Install the primary global cluster, then the standby.Install the Primary and the Standby
8Start the synchronization.Start Synchronization
9Publish the Common Extensions to the primary and the standby.Publish the Common Extensions
10On both global clusters, after the synchronization runs, complete steps 10 to 13 and 18 of the Single Cluster steps, including the placement on the infra nodes.Install the Cluster Plugins, Single Cluster steps
11If no central registry exists, install Harbor on the global cluster or in a dedicated DevOps cluster.Central Registry
12Install the providers that the workload clusters need on the global cluster.Install the Providers on the global Cluster
13Create the workload clusters with the node pools of size A or size B.Cluster Creation Workflow, Choose the Image Registry
14Declare the role labels and taints on the node pools of each workload cluster.Custom Role Nodes
15On each workload cluster, complete steps 11 to 16 and 18 of the Single Cluster steps, including the placement on the infra nodes.Single Cluster steps
—Fail over.Fail Over to the Standby global Cluster
—Upgrade later.Upgrade Overview, Upgrade global Clusters in a DR Environment
  • For disaster recovery, step 2 also covers the keys that both sides share and the network flows between the sides.
  • On size B workload clusters, use the role label and taint of the gateway, monitor, or log pool in place of the infra label and taint when you place each component.

Multi-Cluster with Ceph

When a workload cluster needs storage that follows the Pod across nodes, deploy a small Ceph cluster inside that workload cluster, on a dedicated storage node pool.

Machines and Disks

Node poolNodesCPUMemoryPurpose
storage3 on IaaS; 4 on Bare Metal8 cores24 GiBCeph, on nodes not shared with infra or worker nodes.
  • Spread the storage nodes across failure domains.
  • On Bare Metal, use 4 storage nodes, because a replacement for a failed server can take a long time to arrive. With 3 nodes, Ceph runs degraded on 2 nodes until then, which blocks maintenance and upgrades. With 4 nodes, 3 remain after one failure, and Ceph can restore all 3 replicas.

Sizes are per node, in GB. For what each disk holds and why it has this size, see Disk Configuration.

Diskstorage
IaaS
System disk100
/var/lib/containerd150
/var/lib/kubelet100
/var/cpaas (persistent)50
Bare Metal
System disk, minimum / recommended512 / 512
All environments
Raw disk for the Ceph OSD (SSD)512
Persistent disk for /var/lib/rook100

Changes to the Workload Cluster

ItemChange
Node poolsAdd the storage node pool, with its own role label and taint.
Other components that need persistent volumesUse Ceph block storage (RBD).
Installation stepsInstall Ceph after TopoLVM and before monitoring.

For the installation, see Install.