Manage Nodes on Huawei Cloud Stack
This document explains how to manage worker nodes using Cluster API Machine resources on the Huawei Cloud Stack platform.
TOC
PrerequisitesOverviewNode Management OperationsScaling Worker NodesAdding Worker NodesRemoving Worker NodesAdding a Worker Node GroupChanging Machine InfrastructureUpdating Bootstrap TemplatesVerificationTroubleshootingViewing Controller LogsCommon IssuesPrerequisites
Important Prerequisites
- The control plane must be deployed before performing node operations. See Create Cluster for setup instructions.
- Ensure you have proper access to the HCS platform and required permissions.
- To manage the nodes of the
globalcluster itself, the provider must be installed onglobalfirst; see Manage Nodes on Alauda OS Clusters.
When using the YAML examples in this document, replace only values enclosed in <> with environment-specific values. Preserve the remaining fields unless your cluster policy requires a different value. A cluster created with ac names the MachineDeployment, HCSMachineConfigPool, and machine template of a node group <cluster-name>-<group> and its KubeadmConfigTemplate <cluster-name>-<group>-config, where <group> is computes[].name; the examples use the group worker. A cluster not created with ac may use other names; read them from its MachineDeployment resources.
Overview
Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. A worker node pool is built from four resources:
- Machine Configuration Pool - Network settings for worker nodes
- Machine Template - VM specifications
- Bootstrap Configuration - Node initialization settings
- Machine Deployment - Orchestration of node creation and management
ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page; do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered, so delete the templates that you created yourself.
Node Management Operations
Manage worker nodes with Cluster API manifests. To add another worker node pool to a workload cluster, see Adding a Worker Node Group.
Scaling Worker Nodes
Worker node scaling allows you to adjust cluster capacity based on workload demands.
Adding Worker Nodes
Increase the number of worker nodes to handle increased workload.
Procedure:
-
Check Current Node Status
-
Extend Configuration Pool
Add one machine configuration to the pool for each additional node. Give each new entry the same
persistentDisks[]as the existing entries, including the/var/cpaasdisk.Modify the pool to include new IP entries, then apply:
When you edit the pool, keep all existing
configs[]entries and their acceptedpersistentDisks[]entries unchanged unless you are intentionally appending a new disk slot. -
Scale Up the MachineDeployment
Update the
replicasfield to the desired number of nodes: -
Monitor the Scaling Progress
Continue when the
MachineDeploymentreports the new replica count as ready; otherwise see Troubleshooting.
Removing Worker Nodes
Decrease the number of worker nodes to reduce cluster capacity.
Data Loss Warning
Scaling down removes worker nodes and their ECS instances. Template-owned dataVolumes[] are not preserved. Pool-managed persistent disks declared in HCSMachineConfigPool.spec.configs[].persistentDisks[] remain tracked by the pool and can be reused while the corresponding hostname entry stays in the pool. Ensure:
- Workloads can tolerate node loss through proper replication
- No critical data is stored only on the nodes being removed
- Applications are designed for horizontal scaling
Procedure:
-
Scale Down the MachineDeployment
-
Monitor the Removal Progress
Continue when the machine count matches the new replica count; otherwise see Troubleshooting.
Adding a Worker Node Group
Add workers that have their own addresses, flavor, or availability zone. The global cluster does not support adding a worker node group.
In the following steps, <group> is an existing worker node group and <new-group> is the new one, a DNS-1123 label that no other group uses. control-plane is reserved.
Procedure:
-
Prepare the subnet
Every subnet that a worker uses must be listed in
HCSCluster.spec.network.subnets. To use a subnet that is not listed, create it in the VPC of the cluster first, then add it:Continue when the status is
Trueand the output lists one ID per subnet. -
Read the values of an existing group
The first command prints
<kubernetes-version>,<machine-template-name>, and<bootstrap-template-name>. -
Write the worker objects
- Give the pool at least
replicasentries, one per node, with hostnames and addresses that no other pool uses. Repeat thepersistentDisksentry for each host, and plan it before the node is created; see Change Rendered Objects Later. - Take
imageNamefrom<machine-template-name>, so that the image matches<kubernetes-version>.flavorNameandavailabilityZoneare provider-recognized API values, not the tenant UI display names; every worker of the group uses the selected zone. - Do not set
providerIDorserverId. Put/var/cpaasinpersistentDisks[], not indataVolumes[]. - When
<bootstrap-template-name>has fields that differ fromspec.template.specabove, use its values. For example, on Kubernetes 1.35 and later the kubelet configuration patch needs"imagePullCredentialsVerificationPolicy": "NeverVerify"; earlier versions do not accept this field.
- Give the pool at least
-
Apply the objects and monitor the group
Continue when every
Machineof the group isRunning; then confirm that the nodes areReadyon the cluster.
Changing Machine Infrastructure
To change worker machine specifications (CPU, memory, disk, VM image), create a new machine template and point the MachineDeployment at it. The worker nodes are then replaced one by one. Follow these steps:
Note: HCS dataVolumes[] are not preserved during replacement. To preserve node-local state such as /var/cpaas, declare it in HCSMachineConfigPool.spec.configs[].persistentDisks[] before the rollout, keep MachineDeployment.spec.strategy.rollingUpdate.maxSurge: 0, and leave no free configuration in the pool: a replacement node gets the disks of the node it replaces only when the pool has no free configuration. If your cluster still keeps these disks in HCSMachineTemplate.spec.template.spec.dataVolumes[], first complete Migrate to Pool-Managed Persistent Disks.
-
Create New Machine Template
Copy the existing
HCSMachineTemplateand modify the required values:imageName- VM imageflavorName- Instance typerootVolume.size- System disk sizedataVolumes- Temporary data disk configurations
If you need to add a new pool-managed persistent disk, append it to the worker
HCSMachineConfigPoolfirst. The provider creates or claims the EVS disk, but the running ECS does not mount it until this rolling replacement creates a replacement worker.Then edit
new-template.yamlbefore applying:- Change
metadata.nameto<new-template> - Leave runtime identity fields unset, including
spec.template.spec.providerIDandspec.template.spec.serverId - Remove the server-generated fields:
metadata.resourceVersionmetadata.uidmetadata.generationmetadata.creationTimestampmetadata.managedFields- the
kubectl.kubernetes.io/last-applied-configurationannotation status
-
Deploy New Template
-
Update Machine Deployment
Modify the
MachineDeploymentto reference the new template: -
Monitor Rolling Update
Continue when every machine runs from the new template; otherwise see Troubleshooting.
To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.
In these steps, change only the machine specification. Do not change spec.version of the KubeadmControlPlane or MachineDeployment, and do not switch to a VM image of another Kubernetes version.
Updating Bootstrap Templates
To change the bootstrap configuration of worker nodes (for example kubelet settings or the commands that run when a node joins), create a new KubeadmConfigTemplate and point the MachineDeployment at it. Modifying an existing template in place does not roll out existing machines; only newly created machines pick up the change.
-
Export the Existing Template
-
Modify the Template
Edit
new-bootstrap-template.yaml:- Set
metadata.nameto a new unique name,<new-bootstrap-template> - Remove the server-generated fields listed in Changing Machine Infrastructure
- Edit the fields you want to change. Keep the existing entries of the kubelet patch file
/etc/kubernetes/patches/kubeletconfiguration0+strategic.json, including the certificate paths, and change only what you need - For Kubernetes 1.35 or later, add the required kubelet patch settings to that file
- Set
-
Apply the New Template
-
Update the MachineDeployment
The rolling replacement of the worker nodes then starts:
Continue when every machine uses the new template; otherwise see Troubleshooting.
Verification
After deploying worker nodes, verify the deployment:
Continue when every machine is Running and every node is Ready; otherwise see Troubleshooting.