install-config.yaml Parameters for VMware vSphere
This page describes the objects that ac renders from the install-config.yaml of a VMware vSphere workload cluster, the rules that the provider enforces on them, and the optional topology. For the fields, see Write install-config.yaml.
TOC
Rendered ManifestsMachine Config PoolsProvider RequirementsProvider-managed FieldsOptional TopologySelf-built VIPcontrolPlaneLoadBalancer FieldsAdd a Second NICAdd Data DisksChoose Persistent or EphemeralPersistent Disk LifecyclePod Log DirectoryRendered Manifests
ac writes the objects that it applies to <install-dir>/manifests/, one file per object. All of them are in the cpaas-system namespace. You do not set the Kubernetes or etcd version. ac installs the version described in VM Template and prints it as Pinning Kubernetes <version>, etcd <tag>.
Every VM is a linked clone (cloneMode: linkedClone), unless the template has no snapshot. In that case, the provider makes a full clone. Every VM is powered off with trySoft. For the conditions under which rootDisk.sizeGiB takes effect, see Write install-config.yaml.
Machine Config Pools
Each entry of hosts becomes one slot of the group's VSphereMachineConfigPool. For one worker host with the defaults, the slot renders as follows:
The provider derives the Kubernetes node name, the kubelet serving certificate name, and the kubelet node-ip from the hostname and the primary address of the slot. hostname must be a valid DNS-1123 subdomain. Hostnames and primary IP addresses must be unique across every pool bound to the same Cluster.
Once the status.configStatuses[].state of a slot is InUse or Released, you cannot do any of the following:
- Remove the slot.
- Change its
network.primary.ipor.ipv6. - Remove a disk from
persistentDisks[]. - Change the
sizeGiBormountPathof a persistent disk.
In install-config.yaml, these settings are the ip of the host and its persistent and raw disks.
Do not add a disk to a slot that is InUse or Released. The provider cannot attach the disk to the running node, and it reports errors for that node. An added persistent disk cannot be removed again until the node is replaced. Plan all disks of a slot before the first Machine takes it.
For scale-out and runtime changes, see Manage Nodes on VMware vSphere.
Provider Requirements
ac renders values that satisfy the following rules. Keep to these rules when you edit the objects by hand later, for example during node management. The provider rejects changes to the KubeadmControlPlane and the pools that break the rules. A MachineDeployment that breaks the rules stalls during its next rollout.
- With
spec.replicasof3or more,KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurgeis the integer0. IfmaxSurgeis unset, the Cluster API default1applies, and the provider does not accept it. A single replica keeps the Cluster API default. A cluster with a single control plane node is for testing and cannot be upgraded. MachineDeployment.spec.strategy.rollingUpdate.maxSurgeis0, andmaxUnavailableis at least1. An unset value is not accepted.strategy.type: OnDeleteneeds neither.VSphereMachineConfigPool.spec.configsdeclares at least one slot, and every slot declaresnetworkwith a non-emptynetwork.primary.networkName. Hostnames and primary IP addresses are unique across every pool bound to the sameCluster.- One pool serves exactly one controller. Use one pool for the control plane and one pool per
MachineDeployment. - The provider rejects some changes to a slot that is already
InUseorReleased. See Allocated slot restrictions. - A pool that still owns
InUseslots cannot be deleted.
To confirm that the webhooks are registered in your environment, use the webhook check in Verify the installation. For the rejection messages, see Admission Rejections.
Provider-managed Fields
After the KubeadmControlPlane is applied, the provider sets the CoreDNS and kube-proxy skip annotations to "true" on it. On an ACP 4.5 platform, the platform manages CoreDNS and Kube-OVN in the workload cluster. The provider manages kube-proxy. The kube-proxy repository is acp/k8s from Kubernetes v1.35, and tkestack for earlier versions. It is prefixed with the cpaas.io/registry-address annotation of the Cluster.
Do not remove the CoreDNS and kube-proxy skip annotations, and do not re-apply a stored KubeadmControlPlane manifest that omits them. Otherwise, kubeadm also manages those components, and their images flip back and forth.
Optional Topology
The template in Write install-config.yaml uses one datacenter and one NIC per node. To use a variant, add it to install-config.yaml before you create the cluster. Add one variant at a time. Render the manifests and check the result before you combine variants. After you create the cluster, check the result as described in Verify the Optional Topology.
Self-built VIP
Use this variant to create the cluster with a Self-built VIP instead of an external load balancer. First decide the mode, as described in Control Plane Endpoint.
Meet the Self-built VIP requirements, including the VMware vSphere rules, before you set type: internal. The provider installs Alive, which holds the VIP. The Core Package carries Alive. You do not install it yourself. If it does not install, see Self-built VIP troubleshooting.
controlPlaneLoadBalancer Fields
ac writes VSphereCluster.spec.controlPlaneLoadBalancer from cluster.network.loadBalancer, and only for type: internal. For external, it leaves the field out, because an absent field already means an external LoadBalancer.
The provider accepts spec.controlPlaneLoadBalancer only when the cluster is created. After that:
type,host,port,vrid, andinterfaceare all frozen.- You cannot clear the field.
- You cannot add the field to a cluster that was created without it.
To correct a mistake, you must rebuild the cluster. On multi-NIC nodes, set interface explicitly, because you cannot patch a wrongly auto-detected interface afterwards.
The provider sets ipvs.strictARP: true and adds <vip>/32 to ipvs.excludeCIDRs in the kube-system/kube-proxy configuration of the workload cluster. Do not revert those values by hand, because the provider reapplies them and rolls the kube-proxy DaemonSet again.
To check the VIP, see Verify the Self-built VIP.
Add a Second NIC
Declare the NIC in network.additions of each node group. Give every host one address for each additional NIC in additionalIps, in the same order:
The primary NIC gives the node its identity. The provider derives the kubelet node-ip from it. The additional NICs follow it in the order listed. An additional NIC has an address, a prefix, and the DNS servers of the node, but no gateway.
Add Data Disks
Every node group declares the standard disks, which are all required:
- Control plane nodes (4 disks per node):
var-cpaasinpersistent, plusvar-lib-containerd,var-lib-kubelet, andvar-lib-etcdinephemeral. - Worker nodes (3 disks per node):
var-cpaasinpersistent, plusvar-lib-containerdandvar-lib-kubeletinephemeral.
ac formats them as xfs and mounts them at their standard paths.
For more storage, add rawDisks. Each raw disk becomes a persistent disk with no mount path and no filesystem. It is attached at /dev/disk/by-capv/<name> for TopoLVM or another consumer, as in this example:
On a rolling update, the same VMDK is reattached to the new VM and the symlink is recreated. The disk is never formatted or mounted automatically.
Choose Persistent or Ephemeral
ac places each standard disk in the list it belongs to, and places every raw disk in persistentDisks[]. Within a slot, disk names must be unique across both lists, and so must mount paths. The two lists behave as follows:
/var/cpaas survives VM replacement. /var/lib/kubelet, /var/lib/containerd, and /var/lib/etcd are rebuilt on every new VM.
Persistent Disk Lifecycle
A persistent disk belongs to a slot in one specific VSphereMachineConfigPool, not to the short-lived VM. A slot in another pool is a different allocation, even when it declares the same hostname, IP address, or disk names. The lifecycle is as follows:
- While a Machine uses the slot,
status.configStatuses[].stateisInUseand the VMDKs are attached to its VM. - When the Machine and VM are deleted, the slot becomes
Released. The VMDKs remain available for the replacement. - A replacement Machine of the same pool reuses the
Releasedslot immediately. The provider prefers a released slot over an unused one and reattaches its persistent VMDKs.releaseDelayHoursdoes not delay the reuse. - If no Machine takes the slot before
releaseDelayHoursexpires, the provider makes the slot available again. It then reclaims the persistent VMDKs of the slot, once it confirms that they are detached. Progress appears instatus.persistentDiskStatuses[].phase.
A new pool does not reuse the VMDKs of an earlier pool, even when it has the same name. When you destroy a cluster and create it again, do one of the following:
- Plan datastore capacity for both sets of persistent disks, until the earlier pools finish reclaiming.
- Destroy the cluster with
--reclaim-disks. See Delete a Cluster.
Pool deletion is asynchronous. The finalizer waits for the Machines to go and for the persistent-disk reclaim to finish. Do not remove the finalizer or delete the backing VMDKs by hand to speed it up. Ephemeral VMDKs are deleted with their VM, so they never accumulate across pools.
Read per-disk state from status.persistentDiskStatuses[]. Its phase is one of Creating, Attached, Available, Reclaiming, Reclaimed, or Error. Each entry also reports lastError, retryAfter, taskRef, and the owning machine. status.configStatuses[].lastReleasedTime records when a slot entered Released.
Do not delete a VMDK by hand while its slot is Released, or while its phase is Reclaiming or Error. A replacement Machine may still reuse the VMDK, or the provider may be retrying a reclaim.
Pod Log Directory
When a disk in a slot mounts at /var/lib/containerd, the disk-reconcile script in the guest points /var/log/pods at a directory on that disk. Pod logs then go to that data disk instead of the root filesystem. In the standard layout, that disk is ephemeral, so the relocated pod logs are destroyed with the VM they belong to. Ship any logs that you need to keep to the platform log store.
This relocation is written into the guest only before the first boot. An existing node gets it only when you replace the node. To replace worker nodes, follow Roll out updated worker node configuration. For control plane nodes, use the same steps on the KubeadmControlPlane.