install-config.yaml Parameters for VMware vSphere

This page describes the objects that ac renders from the install-config.yaml of a VMware vSphere workload cluster, the rules that the provider enforces on them, and the optional topology. For the fields, see Write install-config.yaml.

Rendered Manifests

ac writes the objects that it applies to <install-dir>/manifests/, one file per object. All of them are in the cpaas-system namespace. You do not set the Kubernetes or etcd version. ac installs the version described in VM Template and prints it as Pinning Kubernetes <version>, etcd <tag>.

ObjectNameNotes
Cluster<cluster-name>
VSphereCluster<cluster-name>The vCenter server and fingerprint, the credential reference, and the control plane endpoint. For internal, it also holds controlPlaneLoadBalancer.
ConfigMap, Secret, ConfigMap, ClusterResourceSet<cluster-name>-vsphere-cpi-config, -vsphere-cpi-secret, -vsphere-cpi-manifests, -vsphere-cpiThe CPI delivery set.
KubeadmControlPlane, VSphereMachineTemplate, VSphereMachineConfigPool<cluster-name>-control-planemaxSurge: 0 with 3 or more replicas. A single replica gets no rollout strategy. The encryption configuration is a kubeadm file in the KubeadmControlPlane.
MachineDeployment, VSphereMachineTemplate, VSphereMachineConfigPool<cluster-name>-<group>maxSurge: 0, maxUnavailable: 1.
KubeadmConfigTemplate<cluster-name>-<group>-config

Every VM is a linked clone (cloneMode: linkedClone), unless the template has no snapshot. In that case, the provider makes a full clone. Every VM is powered off with trySoft. For the conditions under which rootDisk.sizeGiB takes effect, see Write install-config.yaml.

Machine Config Pools

Each entry of hosts becomes one slot of the group's VSphereMachineConfigPool. For one worker host with the defaults, the slot renders as follows:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineConfigPool
metadata:
  name: <cluster-name>-worker
  namespace: cpaas-system
spec:
  clusterRef:
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: <cluster-name>
  datacenter: <datacenter>
  releaseDelayHours: 24
  configs:
  - hostname: <worker-0-hostname>
    datacenter: <datacenter>
    network:
      primary:
        networkName: <portgroup-name>
        ip: <worker-0-ip>/<prefix-length>
        gateway: <worker-gateway>
        dns:
        - <dns-ip>
    persistentDisks:           # Kept when the VM is replaced
    - name: var-cpaas
      sizeGiB: 100
      mountPath: /var/cpaas
      fsFormat: xfs
      datastore: <datastore>
    ephemeralDisks:            # Recreated with the VM
    - name: var-lib-containerd
      sizeGiB: 100
      mountPath: /var/lib/containerd
      fsFormat: xfs
      datastore: <datastore>
    - name: var-lib-kubelet
      sizeGiB: 100
      mountPath: /var/lib/kubelet
      fsFormat: xfs
      datastore: <datastore>

The provider derives the Kubernetes node name, the kubelet serving certificate name, and the kubelet node-ip from the hostname and the primary address of the slot. hostname must be a valid DNS-1123 subdomain. Hostnames and primary IP addresses must be unique across every pool bound to the same Cluster.

Allocated slot restrictions

Once the status.configStatuses[].state of a slot is InUse or Released, you cannot do any of the following:

  • Remove the slot.
  • Change its network.primary.ip or .ipv6.
  • Remove a disk from persistentDisks[].
  • Change the sizeGiB or mountPath of a persistent disk.

In install-config.yaml, these settings are the ip of the host and its persistent and raw disks.

Do not add a disk to a slot that is InUse or Released. The provider cannot attach the disk to the running node, and it reports errors for that node. An added persistent disk cannot be removed again until the node is replaced. Plan all disks of a slot before the first Machine takes it.

For scale-out and runtime changes, see Manage Nodes on VMware vSphere.

Provider Requirements

ac renders values that satisfy the following rules. Keep to these rules when you edit the objects by hand later, for example during node management. The provider rejects changes to the KubeadmControlPlane and the pools that break the rules. A MachineDeployment that breaks the rules stalls during its next rollout.

  • With spec.replicas of 3 or more, KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge is the integer 0. If maxSurge is unset, the Cluster API default 1 applies, and the provider does not accept it. A single replica keeps the Cluster API default. A cluster with a single control plane node is for testing and cannot be upgraded.
  • MachineDeployment.spec.strategy.rollingUpdate.maxSurge is 0, and maxUnavailable is at least 1. An unset value is not accepted. strategy.type: OnDelete needs neither.
  • VSphereMachineConfigPool.spec.configs declares at least one slot, and every slot declares network with a non-empty network.primary.networkName. Hostnames and primary IP addresses are unique across every pool bound to the same Cluster.
  • One pool serves exactly one controller. Use one pool for the control plane and one pool per MachineDeployment.
  • The provider rejects some changes to a slot that is already InUse or Released. See Allocated slot restrictions.
  • A pool that still owns InUse slots cannot be deleted.

To confirm that the webhooks are registered in your environment, use the webhook check in Verify the installation. For the rejection messages, see Admission Rejections.

Provider-managed Fields

After the KubeadmControlPlane is applied, the provider sets the CoreDNS and kube-proxy skip annotations to "true" on it. On an ACP 4.5 platform, the platform manages CoreDNS and Kube-OVN in the workload cluster. The provider manages kube-proxy. The kube-proxy repository is acp/k8s from Kubernetes v1.35, and tkestack for earlier versions. It is prefixed with the cpaas.io/registry-address annotation of the Cluster.

WARNING

Do not remove the CoreDNS and kube-proxy skip annotations, and do not re-apply a stored KubeadmControlPlane manifest that omits them. Otherwise, kubeadm also manages those components, and their images flip back and forth.

Optional Topology

The template in Write install-config.yaml uses one datacenter and one NIC per node. To use a variant, add it to install-config.yaml before you create the cluster. Add one variant at a time. Render the manifests and check the result before you combine variants. After you create the cluster, check the result as described in Verify the Optional Topology.

Self-built VIP

Use this variant to create the cluster with a Self-built VIP instead of an external load balancer. First decide the mode, as described in Control Plane Endpoint.

Meet the Self-built VIP requirements, including the VMware vSphere rules, before you set type: internal. The provider installs Alive, which holds the VIP. The Core Package carries Alive. You do not install it yourself. If it does not install, see Self-built VIP troubleshooting.

controlPlaneLoadBalancer Fields

ac writes VSphereCluster.spec.controlPlaneLoadBalancer from cluster.network.loadBalancer, and only for type: internal. For external, it leaves the field out, because an absent field already means an external LoadBalancer.

install-config.yamlVSphereCluster.spec.controlPlaneLoadBalancerDescription
type: internaltype: internalInstalls Alive, which holds the VIP.
addresshost, and spec.controlPlaneEndpoint.hostThe Self-built VIP, an IPv4 address.
portport, and spec.controlPlaneEndpoint.portTypically 6443.
vridvridKeepalived virtual_router_id, 1-255. It must be unique in the control plane Layer 2 domain.
interfaceinterfaceThe host interface that carries the VIP, with at most 15 characters. If it is empty, the provider uses the interface that holds the primary IP of the node.
controlPlaneLoadBalancer is write-once

The provider accepts spec.controlPlaneLoadBalancer only when the cluster is created. After that:

  • type, host, port, vrid, and interface are all frozen.
  • You cannot clear the field.
  • You cannot add the field to a cluster that was created without it.

To correct a mistake, you must rebuild the cluster. On multi-NIC nodes, set interface explicitly, because you cannot patch a wrongly auto-detected interface afterwards.

The provider sets ipvs.strictARP: true and adds <vip>/32 to ipvs.excludeCIDRs in the kube-system/kube-proxy configuration of the workload cluster. Do not revert those values by hand, because the provider reapplies them and rolls the kube-proxy DaemonSet again.

To check the VIP, see Verify the Self-built VIP.

Add a Second NIC

Declare the NIC in network.additions of each node group. Give every host one address for each additional NIC in additionalIps, in the same order:

controlPlane:
  network:
    gateway: <cp-gateway>
    dns: [<dns-ip>]
    mask: <prefix-length>
    additions:
      - mask: <nic2-prefix-length>      # Optional: empty takes network.mask
        vsphere:
          name: <nic2-portgroup-name>   # Optional: empty takes provider.vsphere.network.name
          deviceName: eth1              # Optional
  hosts:
    - hostname: <cp-0-hostname>
      ip: <cp-0-ip>
      additionalIps: [<cp-0-nic2-ip>]

The primary NIC gives the node its identity. The provider derives the kubelet node-ip from it. The additional NICs follow it in the order listed. An additional NIC has an address, a prefix, and the DNS servers of the node, but no gateway.

Add Data Disks

Every node group declares the standard disks, which are all required:

  • Control plane nodes (4 disks per node): var-cpaas in persistent, plus var-lib-containerd, var-lib-kubelet, and var-lib-etcd in ephemeral.
  • Worker nodes (3 disks per node): var-cpaas in persistent, plus var-lib-containerd and var-lib-kubelet in ephemeral.

ac formats them as xfs and mounts them at their standard paths.

For more storage, add rawDisks. Each raw disk becomes a persistent disk with no mount path and no filesystem. It is attached at /dev/disk/by-capv/<name> for TopoLVM or another consumer, as in this example:

disks:
  rawDisks:
    - name: data-001
      sizeGiB: 50
  vsphere:                             # Optional: where the data disks go
    datastore: <data-datastore>        # Default: the group's datastore
    storagePolicy: <storage-policy>

On a rolling update, the same VMDK is reattached to the new VM and the symlink is recreated. The disk is never formatted or mounted automatically.

Choose Persistent or Ephemeral

ac places each standard disk in the list it belongs to, and places every raw disk in persistentDisks[]. Within a slot, disk names must be unique across both lists, and so must mount paths. The two lists behave as follows:

persistentDisks[]ephemeralDisks[]
When the VM is deletedDetached and kept, then reattached to the replacement VM at the same slotDeleted with the VM
When the VM is recreatedSame VMDK and same contentAlways a new empty disk, formatted from scratch
Takes part in reclaim and slot releaseYes. While the disk is still attached, it blocks slot release and pool reclaimNo
Observed statestatus.persistentDiskStatuses[], with path, UUID, unit, phase, and owning machinestatus.ephemeralDiskStatuses[], with hostname, name, and unit only
SCSI unitAssigned by the controller at the first clone and reused when the VM is recreatedAssigned by the controller at every clone
Guest addressing/dev/disk/by-capv/<name>, located by VMDK diskUUID, then by SCSI unit/dev/disk/by-capv/<name>, located by SCSI unit only

/var/cpaas survives VM replacement. /var/lib/kubelet, /var/lib/containerd, and /var/lib/etcd are rebuilt on every new VM.

Persistent Disk Lifecycle

A persistent disk belongs to a slot in one specific VSphereMachineConfigPool, not to the short-lived VM. A slot in another pool is a different allocation, even when it declares the same hostname, IP address, or disk names. The lifecycle is as follows:

  1. While a Machine uses the slot, status.configStatuses[].state is InUse and the VMDKs are attached to its VM.
  2. When the Machine and VM are deleted, the slot becomes Released. The VMDKs remain available for the replacement.
  3. A replacement Machine of the same pool reuses the Released slot immediately. The provider prefers a released slot over an unused one and reattaches its persistent VMDKs. releaseDelayHours does not delay the reuse.
  4. If no Machine takes the slot before releaseDelayHours expires, the provider makes the slot available again. It then reclaims the persistent VMDKs of the slot, once it confirms that they are detached. Progress appears in status.persistentDiskStatuses[].phase.

A new pool does not reuse the VMDKs of an earlier pool, even when it has the same name. When you destroy a cluster and create it again, do one of the following:

  • Plan datastore capacity for both sets of persistent disks, until the earlier pools finish reclaiming.
  • Destroy the cluster with --reclaim-disks. See Delete a Cluster.

Pool deletion is asynchronous. The finalizer waits for the Machines to go and for the persistent-disk reclaim to finish. Do not remove the finalizer or delete the backing VMDKs by hand to speed it up. Ephemeral VMDKs are deleted with their VM, so they never accumulate across pools.

Read per-disk state from status.persistentDiskStatuses[]. Its phase is one of Creating, Attached, Available, Reclaiming, Reclaimed, or Error. Each entry also reports lastError, retryAfter, taskRef, and the owning machine. status.configStatuses[].lastReleasedTime records when a slot entered Released.

ac config use-cluster global
ac -n cpaas-system get vspheremachineconfigpool <pool-name> -o yaml
ac -n cpaas-system get events \
  --field-selector involvedObject.kind=VSphereMachineConfigPool,involvedObject.name=<pool-name>
WARNING

Do not delete a VMDK by hand while its slot is Released, or while its phase is Reclaiming or Error. A replacement Machine may still reuse the VMDK, or the provider may be retrying a reclaim.

Pod Log Directory

When a disk in a slot mounts at /var/lib/containerd, the disk-reconcile script in the guest points /var/log/pods at a directory on that disk. Pod logs then go to that data disk instead of the root filesystem. In the standard layout, that disk is ephemeral, so the relocated pod logs are destroyed with the VM they belong to. Ship any logs that you need to keep to the platform log store.

NOTE

This relocation is written into the guest only before the first boot. An existing node gets it only when you replace the node. To replace worker nodes, follow Roll out updated worker node configuration. For control plane nodes, use the same steps on the KubeadmControlPlane.