Manage Nodes on Huawei Cloud Stack

This document explains how to manage worker nodes using Cluster API Machine resources on the Huawei Cloud Stack platform.

Prerequisites

WARNING

Important Prerequisites

  • The control plane must be deployed before performing node operations. See Create Cluster for setup instructions.
  • Ensure you have proper access to the HCS platform and required permissions.
  • To manage the nodes of the global cluster itself, the provider must be installed on global first; see Manage Nodes on Alauda OS Clusters.

When using the YAML examples in this document, replace only values enclosed in <> with environment-specific values. Preserve the remaining fields unless your cluster policy requires a different value. A cluster created with ac names the MachineDeployment, HCSMachineConfigPool, and machine template of a node group <cluster-name>-<group> and its KubeadmConfigTemplate <cluster-name>-<group>-config, where <group> is computes[].name; the examples use the group worker. A cluster not created with ac may use other names; read them from its MachineDeployment resources.

Overview

Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. A worker node pool is built from four resources:

  1. Machine Configuration Pool - Network settings for worker nodes
  2. Machine Template - VM specifications
  3. Bootstrap Configuration - Node initialization settings
  4. Machine Deployment - Orchestration of node creation and management
Clusters created with ac

ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page; do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered, so delete the templates that you created yourself.

Node Management Operations

Manage worker nodes with Cluster API manifests. To add another worker node pool to a workload cluster, see Adding a Worker Node Group.

Scaling Worker Nodes

Worker node scaling allows you to adjust cluster capacity based on workload demands.

Adding Worker Nodes

Increase the number of worker nodes to handle increased workload.

Procedure:

  1. Check Current Node Status

    ac config use-cluster global
    # List all machines in the cluster
    ac get machines.cluster.x-k8s.io -n cpaas-system
    
    # List machines for a specific MachineDeployment
    ac get machines.cluster.x-k8s.io -n cpaas-system -l cluster.x-k8s.io/deployment-name=<cluster-name>-worker
  2. Extend Configuration Pool

    Add one machine configuration to the pool for each additional node. Give each new entry the same persistentDisks[] as the existing entries, including the /var/cpaas disk.

    ac config use-cluster global
    ac get hcsmachineconfigpool <cluster-name>-worker -n cpaas-system -o yaml

    Modify the pool to include new IP entries, then apply:

    ac config use-cluster global
    ac apply -f <updated-pool-config.yaml>

    When you edit the pool, keep all existing configs[] entries and their accepted persistentDisks[] entries unchanged unless you are intentionally appending a new disk slot.

  3. Scale Up the MachineDeployment

    Update the replicas field to the desired number of nodes:

    ac config use-cluster global
    ac patch machinedeployment <cluster-name>-worker -n cpaas-system \
      --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'
  4. Monitor the Scaling Progress

    ac config use-cluster global
    # Watch machines being created
    ac get machines.cluster.x-k8s.io -n cpaas-system -w
    
    # Check MachineDeployment status
    ac get machinedeployment <cluster-name>-worker -n cpaas-system

    Continue when the MachineDeployment reports the new replica count as ready; otherwise see Troubleshooting.

Removing Worker Nodes

Decrease the number of worker nodes to reduce cluster capacity.

WARNING

Data Loss Warning

Scaling down removes worker nodes and their ECS instances. Template-owned dataVolumes[] are not preserved. Pool-managed persistent disks declared in HCSMachineConfigPool.spec.configs[].persistentDisks[] remain tracked by the pool and can be reused while the corresponding hostname entry stays in the pool. Ensure:

  • Workloads can tolerate node loss through proper replication
  • No critical data is stored only on the nodes being removed
  • Applications are designed for horizontal scaling

Procedure:

  1. Scale Down the MachineDeployment

    ac config use-cluster global
    ac patch machinedeployment <cluster-name>-worker -n cpaas-system \
      --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'
  2. Monitor the Removal Progress

    ac config use-cluster global
    ac get machines.cluster.x-k8s.io -n cpaas-system -w

    Continue when the machine count matches the new replica count; otherwise see Troubleshooting.

Adding a Worker Node Group

Add workers that have their own addresses, flavor, or availability zone. The global cluster does not support adding a worker node group.

In the following steps, <group> is an existing worker node group and <new-group> is the new one, a DNS-1123 label that no other group uses. control-plane is reserved.

Procedure:

  1. Prepare the subnet

    Every subnet that a worker uses must be listed in HCSCluster.spec.network.subnets. To use a subnet that is not listed, create it in the VPC of the cluster first, then add it:

    ac config use-cluster global
    ac -n cpaas-system patch hcscluster <cluster-name> --type=json \
      -p='[{"op":"add","path":"/spec/network/subnets/-","value":{"name":"<subnet-name>"}}]'
    ac -n cpaas-system get hcscluster <cluster-name> \
      -o jsonpath='{.status.conditions[?(@.type=="SubnetReady")].status}{"\n"}{.spec.network.subnets[*].id}{"\n"}'

    Continue when the status is True and the output lists one ID per subnet.

  2. Read the values of an existing group

    ac config use-cluster global
    ac -n cpaas-system get machinedeployment <cluster-name>-<group> \
      -o jsonpath='{.spec.template.spec.version}{"\n"}{.spec.template.spec.infrastructureRef.name}{"\n"}{.spec.template.spec.bootstrap.configRef.name}{"\n"}'
    ac -n cpaas-system get hcsmachinetemplate <machine-template-name> -o yaml
    ac -n cpaas-system get kubeadmconfigtemplate <bootstrap-template-name> -o yaml

    The first command prints <kubernetes-version>, <machine-template-name>, and <bootstrap-template-name>.

  3. Write the worker objects

    new-worker-group.yaml
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: HCSMachineConfigPool
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster-name>
      name: <cluster-name>-<new-group>
      namespace: cpaas-system
    spec:
      configs:
      - hostname: <worker-hostname-1>
        networks:
        - subnetName: <subnet-name>
          ipAddress: <worker-ip-1>
        persistentDisks:
        - slot: 0
          size: 100
          type: <disk-type>
          mountPath: /var/cpaas
          format: xfs
    ---
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: HCSMachineTemplate
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster-name>
      name: <cluster-name>-<new-group>
      namespace: cpaas-system
    spec:
      template:
        spec:
          imageName: <vm-image-name>
          flavorName: <instance-flavor>
          availabilityZone: <availability-zone>
          configPoolRef:
            name: <cluster-name>-<new-group>
          rootVolume:
            type: <disk-type>
            size: 100
          dataVolumes:
          - size: 100
            type: <disk-type>
            mountPath: /var/lib/containerd
            format: xfs
          - size: 100
            type: <disk-type>
            mountPath: /var/lib/kubelet
            format: xfs
    ---
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster-name>
      name: <cluster-name>-<new-group>-config
      namespace: cpaas-system
    spec:
      template:
        spec:
          files:
          - content: |
              {
                "apiVersion": "kubelet.config.k8s.io/v1beta1",
                "kind": "KubeletConfiguration",
                "protectKernelDefaults": true,
                "staticPodPath": null,
                "streamingConnectionIdleTimeout": "5m",
                "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
                "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key"
              }
            owner: root:root
            path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
            permissions: "0644"
          joinConfiguration:
            nodeRegistration:
              kubeletExtraArgs:
                protect-kernel-defaults: "true"
                volume-plugin-dir: /opt/libexec/kubernetes/kubelet-plugins/volume/exec/
            patches:
              directory: /etc/kubernetes/patches
          postKubeadmCommands:
          - chmod 600 /var/lib/kubelet/config.yaml
          users:
          - name: boot
            sshAuthorizedKeys:
            - <ssh-public-key>
    ---
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachineDeployment
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster-name>
      name: <cluster-name>-<new-group>
      namespace: cpaas-system
    spec:
      clusterName: <cluster-name>
      replicas: <replicas>
      selector:
        matchLabels:
          cluster.x-k8s.io/cluster-name: <cluster-name>
          pool.name: <cluster-name>-<new-group>
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxSurge: 0
          maxUnavailable: 1
      template:
        metadata:
          labels:
            cluster.x-k8s.io/cluster-name: <cluster-name>
            pool.name: <cluster-name>-<new-group>
        spec:
          clusterName: <cluster-name>
          version: <kubernetes-version>
          nodeDeletionTimeout: 5m
          nodeDrainTimeout: 1m
          bootstrap:
            configRef:
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              kind: KubeadmConfigTemplate
              name: <cluster-name>-<new-group>-config
              namespace: cpaas-system
          infrastructureRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: HCSMachineTemplate
            name: <cluster-name>-<new-group>
            namespace: cpaas-system
    • Give the pool at least replicas entries, one per node, with hostnames and addresses that no other pool uses. Repeat the persistentDisks entry for each host, and plan it before the node is created; see Change Rendered Objects Later.
    • Take imageName from <machine-template-name>, so that the image matches <kubernetes-version>. flavorName and availabilityZone are provider-recognized API values, not the tenant UI display names; every worker of the group uses the selected zone.
    • Do not set providerID or serverId. Put /var/cpaas in persistentDisks[], not in dataVolumes[].
    • When <bootstrap-template-name> has fields that differ from spec.template.spec above, use its values. For example, on Kubernetes 1.35 and later the kubelet configuration patch needs "imagePullCredentialsVerificationPolicy": "NeverVerify"; earlier versions do not accept this field.
  4. Apply the objects and monitor the group

    ac config use-cluster global
    ac apply -f new-worker-group.yaml
    ac -n cpaas-system get machines.cluster.x-k8s.io \
      -l cluster.x-k8s.io/deployment-name=<cluster-name>-<new-group> -w

    Continue when every Machine of the group is Running; then confirm that the nodes are Ready on the cluster.

Changing Machine Infrastructure

To change worker machine specifications (CPU, memory, disk, VM image), create a new machine template and point the MachineDeployment at it. The worker nodes are then replaced one by one. Follow these steps:

Note: HCS dataVolumes[] are not preserved during replacement. To preserve node-local state such as /var/cpaas, declare it in HCSMachineConfigPool.spec.configs[].persistentDisks[] before the rollout, keep MachineDeployment.spec.strategy.rollingUpdate.maxSurge: 0, and leave no free configuration in the pool: a replacement node gets the disks of the node it replaces only when the pool has no free configuration. If your cluster still keeps these disks in HCSMachineTemplate.spec.template.spec.dataVolumes[], first complete Migrate to Pool-Managed Persistent Disks.

  1. Create New Machine Template

    Copy the existing HCSMachineTemplate and modify the required values:

    • imageName - VM image
    • flavorName - Instance type
    • rootVolume.size - System disk size
    • dataVolumes - Temporary data disk configurations

    If you need to add a new pool-managed persistent disk, append it to the worker HCSMachineConfigPool first. The provider creates or claims the EVS disk, but the running ECS does not mount it until this rolling replacement creates a replacement worker.

    ac config use-cluster global
    ac get hcsmachinetemplate <current-template> -n cpaas-system -o yaml > new-template.yaml

    Then edit new-template.yaml before applying:

    • Change metadata.name to <new-template>
    • Leave runtime identity fields unset, including spec.template.spec.providerID and spec.template.spec.serverId
    • Remove the server-generated fields:
      • metadata.resourceVersion
      • metadata.uid
      • metadata.generation
      • metadata.creationTimestamp
      • metadata.managedFields
      • the kubectl.kubernetes.io/last-applied-configuration annotation
      • status
  2. Deploy New Template

    ac config use-cluster global
    ac apply -f new-template.yaml -n cpaas-system
  3. Update Machine Deployment

    Modify the MachineDeployment to reference the new template:

    ac config use-cluster global
    ac patch machinedeployment <cluster-name>-worker -n cpaas-system \
      --type='merge' -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":"<new-template>"}}}}}'
  4. Monitor Rolling Update

    ac config use-cluster global
    ac get machines.cluster.x-k8s.io -n cpaas-system -w

    Continue when every machine runs from the new template; otherwise see Troubleshooting.

To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.

In these steps, change only the machine specification. Do not change spec.version of the KubeadmControlPlane or MachineDeployment, and do not switch to a VM image of another Kubernetes version.

Updating Bootstrap Templates

To change the bootstrap configuration of worker nodes (for example kubelet settings or the commands that run when a node joins), create a new KubeadmConfigTemplate and point the MachineDeployment at it. Modifying an existing template in place does not roll out existing machines; only newly created machines pick up the change.

  1. Export the Existing Template

    ac config use-cluster global
    ac get kubeadmconfigtemplate <cluster-name>-worker-config -n cpaas-system -o yaml > new-bootstrap-template.yaml
  2. Modify the Template

    Edit new-bootstrap-template.yaml:

    • Set metadata.name to a new unique name, <new-bootstrap-template>
    • Remove the server-generated fields listed in Changing Machine Infrastructure
    • Edit the fields you want to change. Keep the existing entries of the kubelet patch file /etc/kubernetes/patches/kubeletconfiguration0+strategic.json, including the certificate paths, and change only what you need
    • For Kubernetes 1.35 or later, add the required kubelet patch settings to that file
  3. Apply the New Template

    ac config use-cluster global
    ac apply -f new-bootstrap-template.yaml -n cpaas-system
  4. Update the MachineDeployment

    ac config use-cluster global
    ac patch machinedeployment <cluster-name>-worker -n cpaas-system \
      --type='merge' -p='{"spec":{"template":{"spec":{"bootstrap":{"configRef":{"name":"<new-bootstrap-template>"}}}}}}'

    The rolling replacement of the worker nodes then starts:

    ac config use-cluster global
    ac get machines.cluster.x-k8s.io -n cpaas-system -w

    Continue when every machine uses the new template; otherwise see Troubleshooting.

Verification

After deploying worker nodes, verify the deployment:

ac config use-cluster global
# Check machine status
ac get machines.cluster.x-k8s.io -n cpaas-system

# Check MachineDeployment status
ac get machinedeployment -n cpaas-system

# Verify nodes are Ready
ac config sync
ac config use-cluster <cluster-name>
ac get nodes
ac config use-cluster global

Continue when every machine is Running and every node is Ready; otherwise see Troubleshooting.

Troubleshooting

Viewing Controller Logs

ac config use-cluster global

# View HCS controller logs
ac logs -n cpaas-system deployment/cluster-api-provider-hcs-manager

# View machine details
ac describe hcsmachine <machine-name> -n cpaas-system

Common Issues

SymptomAction
A node fails to join the clusterConfirm that the image matches the Kubernetes version, that the nodes can reach each other, and that the configuration pool has unused entries.
A machine is stuck in provisioningRead the controller logs for the error. Confirm that HCS has the resources and that the credentials and permissions are valid.