Manage Nodes on VMware vSphere

This document explains how to manage worker nodes on VMware vSphere after the baseline cluster is running. Node lifecycle operations are managed through VSphereMachineConfigPool, VSphereMachineTemplate, KubeadmConfigTemplate, and MachineDeployment resources.

To add another worker node group to a workload cluster, see Add a worker node group.

Prerequisites

Before you begin, ensure the following conditions are met:

  • The workload cluster was created successfully. See Create a Cluster on VMware vSphere.
  • The worker machine config pool has enough available slots.
  • The control plane is healthy and reachable.
  • You know which manifest files currently define the worker nodes.
  • You know the rollout and pool rules in Provider Requirements.
  • To manage the nodes of the global cluster itself, the provider must be installed on global first; see Manage Nodes on Alauda OS Clusters.
WARNING

A Day-2 apply can be rejected

The rollout and pool rules apply to every edit this page makes:

  • Re-applying an archived MachineDeployment manifest re-submits the whole object, including an apply whose only intended change is replicas. Before you apply it, confirm the file sets strategy.rollingUpdate.maxSurge: 0 and a maxUnavailable of at least 1, or the apply is rejected. strategy.type: OnDelete needs neither.
  • A KubeadmControlPlane must set spec.replicas to 1, or to 3 or more with spec.rolloutStrategy.rollingUpdate.maxSurge: 0, or an edit to it is rejected.
  • A slot whose status.configStatuses[].state is InUse or Released cannot be removed, cannot have its network.primary.ip or .ipv6 changed, and cannot lose a disk from persistentDisks[] or change one's sizeGiB or mountPath. Adding a slot is accepted. Do not add a disk to such a slot; see Allocated slot restrictions.

Match a rejection message against Admission rejections.

Clusters created with ac

ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page; do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered, so delete the templates that you created yourself.

Steps

Scale out worker nodes

When you add more worker nodes, update the worker machine config pool before you increase the replica count.

Worker scale-out depends on the relationship between MachineDeployment.spec.replicas and the available node slots in VSphereMachineConfigPool.spec.configs[]. The number of slots can be greater than replicas, and idle slots do not affect a running cluster. If replicas exceeds the number of available slots, CAPV cannot assign the new workers correctly.

  1. Add one or more new node slots to the worker VSphereMachineConfigPool, <cluster_name>-<group> for a cluster created with ac.
  2. Update replicas in the MachineDeployment of the same name.
  3. Add the slots first, then raise the replicas, as shown below.

The following example adds a worker slot with the required data disks:

- hostname: "<worker_node_name_2>"
  datacenter: "<worker_02_datacenter>"
  network:
    primary:
      networkName: "<nic1_network_name>"
      ip: "<worker_02_nic1_ip>/<nic1_prefix>"
      gateway: "<nic1_gateway>"
      dns:
      - "<nic1_dns_1>"
  persistentDisks:
  - name: var-cpaas
    sizeGiB: <worker_var_cpaas_size_gib>
    mountPath: /var/cpaas
    fsFormat: ext4
  ephemeralDisks:
  - name: var-lib-kubelet
    sizeGiB: <worker_var_lib_kubelet_size_gib>
    mountPath: /var/lib/kubelet
    fsFormat: ext4
  - name: var-lib-containerd
    sizeGiB: <worker_var_lib_containerd_size_gib>
    mountPath: /var/lib/containerd
    fsFormat: ext4

The example uses the disk layout of a cluster created on provider v1.0.17. Match fsFormat to the existing slots: ac formats the standard disks as xfs. A pool created on an earlier provider has no ephemeralDisks[] and declares /var/cpaas, /var/lib/containerd, and /var/lib/etcd in persistentDisks[]; match the slots that pool already has instead of introducing ephemeralDisks[] into it.

To attach an additional raw disk without formatting or mounting, omit mountPath and fsFormat:

  persistentDisks:
  # ...standard persistent disks...
  - name: app-data
    sizeGiB: 50

The raw disk is available inside the guest OS at /dev/disk/by-capv/app-data. On rolling updates, the same VMDK is re-attached to the replacement VM and the symlink is recreated. The application remains responsible for the disk contents and runtime mount.

Then increase the worker replicas:

replicas: <worker_replicas>

Use the following order:

ac config use-cluster global
ac -n cpaas-system edit vspheremachineconfigpool <cluster_name>-<group>   # add the slots first
ac -n cpaas-system scale machinedeployment <cluster_name>-<group> --replicas=<worker_replicas>

Note: Keep the same existing VSphereMachineConfigPool.metadata.name when scale-out or replacement must preserve the logical pool. Slots and VMDKs are not reused across differently named pool objects. releaseDelayHours is the reclaim grace period for an unused Released slot, not a delay before same-pool reuse.

Add a worker node group

Add workers that have their own slots, VM specification, or placement. The global cluster does not support adding a worker node group.

In the following steps, <group> is an existing worker node group and <new_group> is the new one, a DNS-1123 label that no other group uses. control-plane is reserved.

  1. Read the values of an existing group:

    ac config use-cluster global
    ac -n cpaas-system get machinedeployment <cluster_name>-<group> \
      -o jsonpath='{.spec.template.spec.version}{"\n"}{.spec.template.spec.infrastructureRef.name}{"\n"}{.spec.template.spec.bootstrap.configRef.name}{"\n"}'
    ac -n cpaas-system get vspheremachinetemplate <machine_template_name> -o yaml
    ac -n cpaas-system get kubeadmconfigtemplate <bootstrap_template_name> -o yaml

    The first command prints <k8s_version>, <machine_template_name>, and <bootstrap_template_name>.

  2. Write the worker objects:

    new-worker-group.yaml
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineConfigPool
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster_name>
      name: <cluster_name>-<new_group>
      namespace: cpaas-system
    spec:
      clusterRef:
        apiVersion: cluster.x-k8s.io/v1beta1
        kind: Cluster
        name: <cluster_name>
      datacenter: <datacenter>
      releaseDelayHours: 24
      configs:
      - hostname: <worker_node_name_1>
        datacenter: <datacenter>
        network:
          primary:
            deviceName: eth0
            networkName: <nic1_network_name>
            ip: <worker_01_nic1_ip>/<nic1_prefix>
            gateway: <nic1_gateway>
            dns:
            - <nic1_dns_1>
        persistentDisks:
        - name: var-cpaas
          datastore: <worker_data_datastore>
          sizeGiB: <worker_var_cpaas_size_gib>
          mountPath: /var/cpaas
          fsFormat: xfs
        ephemeralDisks:
        - name: var-lib-containerd
          datastore: <worker_data_datastore>
          sizeGiB: <worker_var_lib_containerd_size_gib>
          mountPath: /var/lib/containerd
          fsFormat: xfs
        - name: var-lib-kubelet
          datastore: <worker_data_datastore>
          sizeGiB: <worker_var_lib_kubelet_size_gib>
          mountPath: /var/lib/kubelet
          fsFormat: xfs
    ---
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineTemplate
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster_name>
      name: <cluster_name>-<new_group>
      namespace: cpaas-system
    spec:
      template:
        spec:
          server: <vsphere_server>
          datacenter: <datacenter>
          template: <template_name>
          cloneMode: linkedClone
          folder: <vm_folder>
          resourcePool: <resource_pool>
          datastore: <worker_system_datastore>
          diskGiB: <worker_system_disk_gib>
          memoryMiB: <worker_memory_mib>
          numCPUs: <worker_num_cpus>
          os: Linux
          powerOffMode: trySoft
          network:
            devices:
            - networkName: <nic1_network_name>
          machineConfigPoolRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: VSphereMachineConfigPool
            name: <cluster_name>-<new_group>
            namespace: cpaas-system
    ---
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster_name>
      name: <cluster_name>-<new_group>-config
      namespace: cpaas-system
    spec:
      template:
        spec:
          files:
          - content: |
              {
                "apiVersion": "kubelet.config.k8s.io/v1beta1",
                "kind": "KubeletConfiguration",
                "protectKernelDefaults": true,
                "staticPodPath": null,
                "streamingConnectionIdleTimeout": "5m",
                "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
                "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key"
              }
            owner: root:root
            path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
            permissions: "0644"
          - content: |
              #!/bin/bash
              set -euo pipefail
              until mountpoint -q /var/lib/containerd; do
                echo "waiting for /var/lib/containerd mount"
                sleep 1
              done
              systemctl restart containerd
              until systemctl is-active --quiet containerd; do
                echo "waiting for containerd"
                sleep 1
              done
              if [ ! -d "/root/images" ]; then
                echo "ERROR: /root/images directory not found" >&2
                exit 1
              fi
              image_count=0
              for image_file in /root/images/*.tar; do
                if [ -f "$image_file" ]; then
                  echo "importing image: $image_file"
                  ctr -n k8s.io images import "$image_file"
                  image_count=$((image_count + 1))
                fi
              done
              if [ "$image_count" -eq 0 ]; then
                echo "ERROR: no tar files found in /root/images" >&2
                exit 1
              fi
              echo "imported $image_count images"
            owner: root:root
            path: /usr/local/bin/capv-load-local-images.sh
            permissions: "0755"
          - content: |
              nameserver <nic1_dns_1>
            owner: root:root
            path: /etc/resolv.conf
            permissions: "0644"
          joinConfiguration:
            nodeRegistration:
              criSocket: /var/run/containerd/containerd.sock
              ignorePreflightErrors:
              - ImagePull
              kubeletExtraArgs:
                cloud-provider: external
                volume-plugin-dir: /opt/libexec/kubernetes/kubelet-plugins/volume/exec/
              name: '{{ local_hostname }}'
            patches:
              directory: /etc/kubernetes/patches
          postKubeadmCommands:
          - chmod 600 /var/lib/kubelet/config.yaml
          preKubeadmCommands:
          - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
          - echo "::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6" >/etc/hosts
          - echo "127.0.0.1   {{ ds.meta_data.hostname }} {{ local_hostname }} localhost localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts
          - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started"
          - /usr/local/bin/capv-load-local-images.sh
          users:
          - name: boot
            sshAuthorizedKeys:
            - <ssh_public_key>
            sudo: ALL=(ALL) NOPASSWD:ALL
    ---
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachineDeployment
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <cluster_name>
      name: <cluster_name>-<new_group>
      namespace: cpaas-system
    spec:
      clusterName: <cluster_name>
      replicas: <worker_replicas>
      selector:
        matchLabels:
          cluster.x-k8s.io/cluster-name: <cluster_name>
          pool.name: <cluster_name>-<new_group>
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxSurge: 0
          maxUnavailable: 1
      template:
        metadata:
          labels:
            cluster.x-k8s.io/cluster-name: <cluster_name>
            pool.name: <cluster_name>-<new_group>
        spec:
          clusterName: <cluster_name>
          version: <k8s_version>
          nodeDeletionTimeout: 5m
          nodeDrainTimeout: 1m
          bootstrap:
            configRef:
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              kind: KubeadmConfigTemplate
              name: <cluster_name>-<new_group>-config
              namespace: cpaas-system
          infrastructureRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: VSphereMachineTemplate
            name: <cluster_name>-<new_group>
            namespace: cpaas-system
    • Give the pool at least replicas slots. Hostnames and primary IP addresses must be unique across every pool of the cluster, and each MachineDeployment needs its own pool; see Provider Requirements.
    • Take template from <machine_template_name>, so that the VM template matches <k8s_version>. Take server, folder, and resourcePool from it as well.
    • Keep maxSurge: 0: a rollout with the Cluster API default stalls on a full pool.
    • When an existing object has fields that differ from the example, use its values: for example, fsFormat and the disks of a slot, a second NIC, or a failure domain. When <bootstrap_template_name> differs from spec.template.spec above, use its values too. For example, on Kubernetes 1.35 and later the kubelet configuration patch needs "imagePullCredentialsVerificationPolicy": "NeverVerify"; earlier versions do not accept this field.
  3. Apply the objects and monitor the group:

    ac config use-cluster global
    ac apply -f new-worker-group.yaml
    ac -n cpaas-system get machines.cluster.x-k8s.io \
      -l cluster.x-k8s.io/deployment-name=<cluster_name>-<new_group> -w

    Continue when every Machine of the group is Running; then confirm that the nodes are Ready on the cluster. Match a rejection message against Admission rejections.

Roll out updated worker node configuration

When you need to change worker VM specifications, create a new VSphereMachineTemplate and update the MachineDeployment to reference it. This triggers a rolling update that replaces worker nodes with the new configuration.

Typical changes include:

  • VM template name (spec.template.spec.template)
  • CPU or memory sizing (numCPUs, memoryMiB)
  • System disk sizing (diskGiB), on a full clone only: a linked clone ignores it
  • VM network devices (network.devices)

In these steps, change only the machine specification. Do not change spec.version of the KubeadmControlPlane or MachineDeployment, and do not switch to a VM template of another Kubernetes version.

Persistent disks are declared in VSphereMachineConfigPool slots rather than in VSphereMachineTemplate, so a new template does not change them. A slot that is InUse or Released rejects changing an existing disk's sizeGiB or mountPath, so a different geometry for an existing disk needs a new slot, not an edit of the current one. See Allocated slot restrictions.

  1. Export the existing template

    ac config use-cluster global
    ac get vspheremachinetemplate <cluster_name>-worker -n <namespace> -o yaml > new-worker-template.yaml
  2. Modify the template

    Edit new-worker-template.yaml:

    • Set metadata.name to a new unique name (for example, <cluster_name>-worker-v2)
    • Update the desired specification fields
    • Remove server-generated fields: metadata.resourceVersion, metadata.uid, metadata.generation, metadata.creationTimestamp, metadata.managedFields, metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], and status
  3. Apply the new template

    ac config use-cluster global
    ac apply -f new-worker-template.yaml
  4. Update the MachineDeployment reference

    ac config use-cluster global
    ac patch machinedeployment <cluster_name>-worker -n <namespace> \
      --type='merge' -p='{
        "spec": {
          "template": {
            "spec": {
              "infrastructureRef": {
                "name": "<new-template-name>"
              }
            }
          }
        }
      }'

    If you also need to change bootstrap settings, see Updating Bootstrap Templates below.

  5. Monitor the rolling update

    ac config use-cluster global
    ac -n <namespace> get machinedeployment <cluster_name>-worker -w
    ac -n <namespace> get machine

To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.

TIP

Rolling back a failed update

If the rolling update fails (for example, new VMs fail to boot), revert the MachineDeployment reference back to the previous template name. This works only while the Kubernetes minor version is unchanged. The old template still exists and Cluster API will roll back to it.

Verify worker node status

Run the following commands to verify the management-cluster and workload-cluster status:

ac config use-cluster global
ac -n <namespace> get machinedeployment,machine,vspheremachine,vspherevm
ac config sync
ac config use-cluster <cluster_name>
ac get nodes -o wide
ac config use-cluster global

Confirm the following results:

  • The target worker replica count is reached.
  • Every new worker node joins the cluster.
  • The nodes eventually become Ready.

Updating Bootstrap Templates

To update bootstrap configuration, create a new KubeadmConfigTemplate and update the MachineDeployment reference.

  1. Export the existing template

    ac config use-cluster global
    ac get kubeadmconfigtemplate <cluster_name>-worker-config -n <namespace> -o yaml > new-bootstrap-template.yaml
  2. Modify the template

    Edit new-bootstrap-template.yaml:

    • Set metadata.name to a new unique name (for example, <cluster_name>-worker-config-v2)
    • Update the desired bootstrap configuration fields
    • For Kubernetes 1.35 or later, add the required kubelet patch settings to /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
    • Remove the same server-generated fields listed in Roll out updated worker node configuration step 2
  3. Apply the new template

    ac config use-cluster global
    ac apply -f new-bootstrap-template.yaml
  4. Update the MachineDeployment reference

    ac config use-cluster global
    ac patch machinedeployment <cluster_name>-worker -n <namespace> \
      --type='merge' -p='{
        "spec": {
          "template": {
            "spec": {
              "bootstrap": {
                "configRef": {
                  "name": "<new-bootstrap-template-name>"
                }
              }
            }
          }
        }
      }'

    The Cluster API controller triggers a rolling update. Existing machines continue using the old bootstrap configuration until they are replaced.

Runtime Topology Changes

Networking, placement, and disk topology are part of the machine templates and machine config pool. Treat a change to these fields as a rollout of replacement machines, not as an in-place VM edit:

  1. Add the required slot definitions to the existing VSphereMachineConfigPool before increasing replicas or starting replacement. An allocated slot rejects some changes; see the warning in Prerequisites.
  2. Create a new VSphereMachineTemplate when its VM, network, placement, or system-disk fields change.
  3. Create a new KubeadmConfigTemplate when bootstrap files or node configuration change.
  4. Update the MachineDeployment references only after the new pool capacity and templates exist.
  5. Monitor replacement Machines and confirm that every node becomes Ready before removing old templates.

For the exact second-NIC and persistent-disk fields, use Optional Topology in Create a Cluster on VMware vSphere. Preserve the same cross-resource relationships during Day-2 changes:

  • Keep VSphereMachineConfigPool.spec.configs[].network.additional, and VSphereMachineTemplate.spec.template.spec.network.devices consistent when NIC topology changes.
  • Do not remove the required var-cpaas persistent disk, or the required var-lib-kubelet and var-lib-containerd ephemeral disks, or the control-plane var-lib-etcd ephemeral disk.
  • Keep a new slot's disk lists identical to the slots already in that pool. ephemeralDisks[] exists from provider v1.0.17, but a pool created on an earlier provider declares /var/cpaas, /var/lib/containerd, and /var/lib/etcd in persistentDisks[]; do not convert those slots, and do not mix layouts inside one pool.
  • Do not reshape or add disks to an allocated slot. A slot that is InUse or Released rejects removing a persistent disk or changing its sizeGiB or mountPath. Resizing means planning a new slot.
  • Do not treat machineNamingStrategy, wipeFilesystem, or releaseDelayHours as a switch that moves disks between pools or accelerates datastore cleanup.

When you expand from one NIC to two NICs, update all of the following fields together:

  1. Append the second NIC entry to VSphereMachineConfigPool.spec.configs[].network.additional; keep network.primary unchanged.
  2. Add the second device to VSphereMachineTemplate.spec.template.spec.network.devices in the new template.

When you revert from two NICs to one NIC, remove the corresponding second entry from both fields. Leave network.additional empty or remove the key entirely, and roll out a new VSphereMachineTemplate; do not edit existing VMs directly.

Troubleshooting

Use the following checks first when worker node management fails:

  • Check VSphereMachine conditions for MachineConfigPoolReady. If False, the reason indicates why slot allocation failed:
    • PoolBoundToOtherConsumer: the pool is already bound to a different KubeadmControlPlane or MachineDeployment.
    • NoAvailableSlots: no slot matches the required datacenter.
  • Verify that the worker machine config pool still has free slots.
  • Verify that the worker IP addresses, gateway, and DNS settings are correct.
  • Verify that the worker VM template still matches the required Kubernetes version and guest-tools requirements.
  • Check VSphereVM.status.addresses when a node is waiting for IP allocation.

Next Steps

For cluster creation and persistent-disk diagnostics, see Create a Cluster on VMware vSphere and its Persistent disk lifecycle.