#Troubleshooting VMware vSphere Workload Clusters
Start from the stage that ac adm alpha cluster status reports. For failures that are common to all providers, see Troubleshooting.
#Creation Stages
| Symptom | Where to look first |
|---|---|
create reports a vCenter object that it cannot find | Fix the name in install-config.yaml and run create again. ac reports only one missing object per run. Names given as inventory paths (they contain /) are checked only when the VMs are cloned. |
[1/4] CAPI Cluster does not advance | The Cluster and VSphereCluster conditions. |
[2/4] KubeadmControlPlane does not advance | The KubeadmControlPlane, Machine, VSphereMachine, and VSphereVM conditions, and the pool conditions and status.configStatuses. |
| The control plane does not come up | The endpoint and the registry: the health of the load balancer or VIP, DNS, the registry API, the ACP port rules, and whether the global cluster and the nodes can reach them. |
| Nodes stay uninitialized | The CPI delivery: the ClusterResourceSetBinding and the CPI Pods in the workload cluster. |
[3/4] MachineDeployment does not advance | The MachineDeployment rollout, the VM status, the CPI, and the CNI. |
[4/4] ClusterView does not advance | The platform ClusterView, the registry API, and the provider manager log. |
Collect the state first. Run the following commands from a machine where ac is logged in to the platform:
ac config use-cluster global
ac -n cpaas-system describe clusters.cluster.x-k8s.io <cluster-name>
ac -n cpaas-system describe vspherecluster <cluster-name>
ac -n cpaas-system describe kubeadmcontrolplane <cluster-name>-control-plane
ac -n cpaas-system describe machinedeployment <cluster-name>-<group>
ac -n cpaas-system get clusters.cluster.x-k8s.io,vspherecluster,kubeadmcontrolplane,machinedeployment,machine,vspheremachine,vspherevm
ac -n cpaas-system logs deploy/capi-controller-managerIf the network plugin is not installed, also inspect the logs of the platform network controller and cluster lifecycle controller.
#Common Causes
| Result | Action |
|---|---|
| The CPI resources are not delivered | Check that the ClusterResourceSet and its ClusterResourceSetBinding exist, as in CPI and Pools. If the ClusterResourceSet exists and nothing is applied, contact technical support. |
A ClusterResourceSet exists, but no ClusterResourceSetBinding is created | Check that the controller has delete permission on the referenced ConfigMap and Secret. |
| The network plugin is not installed | Check that the cluster annotations are present and that the platform controllers processed them. |
The cpaas.io/registry-address annotation is missing or wrong | Set it to the permanent <registry_address>, without a scheme or /v2/. Then check cpaas-system/public-registry-credential. |
A machine is stuck in Provisioning | Read the MachineConfigPoolReady condition of the VSphereMachine. Then read the pool conditions MembersValid, MembersUnique, PersistentDisksReady, ClusterRefReady, and VCenterAvailable. |
A machine never bootstraps; BootstrapReady is False with BootstrapSecretGetFailed or BootstrapSecretContentInvalid | Inspect the KubeadmConfig of that Machine and its bootstrap data Secret before you look at vCenter. |
| A node was powered off in vCenter and stays off | Power the VM on in vCenter. The provider does not power on a VM that was powered off after its first boot. You do not need a rollout. |
VSphereMachine clears Ready without an obvious cause | Read the VSphereVM conditions. Ready is cleared while the VM, its providerID, or its network information is unavailable. |
| A VM waits for an IP address | Check VMware Tools, the static IP settings, and VSphereVM.status.addresses. |
Workload Node objects have no spec.providerID | Check the CPI delivery first. Then look for an old VM with the same guest hostname in the same datacenter. Fix or remove the duplicate, and then restart the CPI Pods as shown below this table. |
| The datastore runs out of space | List every VSphereMachineConfigPool, including terminating ones. Each pool owns its own persistent disks, so repeated deployments accumulate sets of disks. Read status.configStatuses[], status.persistentDiskStatuses[] (phase, lastError, retryAfter, taskRef), and the pool Events. Do not delete a VMDK, remove a pool finalizer, or edit status.persistentDiskStatuses[] by hand. If reclaim does not progress, contact technical support. See Persistent disk lifecycle. |
The system disk keeps the template's size although rootDisk.sizeGiB is set | The VM is a linked clone, and rootDisk.sizeGiB takes effect only when the template has no snapshot (a full clone). See the global cluster template. |
| The control plane endpoint does not come up | Check TCP 6443 passthrough, all control plane backends, HTTPS /healthz, DNS and certificate SANs, and reachability from the global cluster and the control plane nodes. |
| The TLS connection to vCenter fails | Check the thumbprint, the vCenter address, and any proxy settings. |
After you fix a duplicate hostname, restart the CPI Pods:
ac config sync
ac config use-cluster <cluster-name>
ac -n kube-system delete pod -l k8s-app=vsphere-cloud-controller-manager
ac config use-cluster global#Self-built VIP
Read the SelfBuiltLoadBalancerReady condition on the VSphereCluster, as in Verify the Self-built VIP. Then find the result in the following table:
| Result | Action |
|---|---|
False, reason SelfBuiltLoadBalancerReconciling | The VIP is still being set up. Check again later. |
False, reason SelfBuiltLoadBalancerNotReady | Check the Alive Pods in kube-system and the alive AppRelease in cpaas-system of the workload cluster. Also check that the VIP is in the control plane Layer 2 domain, that the vrid is unique, and that VRRP and gratuitous ARP are allowed. |
False, reason InvalidSelfBuiltLoadBalancerConfiguration | The loadBalancer values are invalid. You cannot change them on an existing cluster. Delete the cluster and create it again with corrected values. |
healthz through the VIP does not print ok, and the cluster cannot be reached through the VIP | Check that the VIP is in the control plane Layer 2 domain, that the vrid is unique, and that VRRP and gratuitous ARP are allowed. Then check the Alive Pods. |
No Alive Pods and no alive AppRelease | Check the Alive plugin on the global cluster with the commands below. |
ac config use-cluster global
ac get clustermodule global -o jsonpath='{.spec.version}{"\n"}'
ac get moduleplugin alive -o jsonpath='{.status.targetClusterVersions}{"\n"}'
ac get moduleplugin alive -o jsonpath='{.status.latestVersion}{"\n"}'
ac get moduleconfig alive-<alive-version> -o jsonpath='{.status.readyForDeploy}{"\n"}'Replace <alive-version> with the targetClusterVersions entry for the version that the first command prints. If there is no such entry, use latestVersion. Continue when the last command prints true. Otherwise, the Core Package on this platform is incomplete; import the complete Core Package of your platform release.
#Admission Rejections
The provider can reject an apply or edit before anything is created. Find a fragment of the rejection message in the first column of the following table. For the rules, see Provider Requirements.
| Rejection contains | Cause and fix |
|---|---|
maxSurge must be 0 when the infrastructure template references a machineConfigPoolRef | Set maxSurge to the integer 0 on the KubeadmControlPlane or MachineDeployment. If you leave it unset, the Cluster API default 1 applies. |
maxUnavailable must be at least 1 | Set maxUnavailable to 1 or more on the MachineDeployment. If you leave it unset, the Cluster API default 0 applies. |
must have 1 or at least 3 replicas | The control plane has 2 replicas. Use 1, or 3 or more with maxSurge: 0. |
also used by pool | A hostname or primary IP is already used by a slot in another pool that is bound to the same Cluster in this namespace. |
allocated slot | The slot is InUse or Released. The message names what is frozen: the primary IP or IPv6, or the sizeGiB or mountPath of a persistent disk. Or it reports that you cannot remove the slot or one of its persistent disks. See Allocated slot restrictions. |
cannot delete pool while | The pool still owns InUse slots. Delete the Machines that hold them first. |
is already referenced by or is bound to | Two controllers reference one pool. Give each KubeadmControlPlane and MachineDeployment its own pool. |
cannot change clusterRef while consumerRef is set | The pool is already bound to a controller. Create a new pool instead of changing the cluster that this pool refers to. |
VSphereMachineTemplate spec.template.spec field is immutable | Create a new template under a new name instead of editing the existing one. See Roll out updated worker node configuration. |
is immutable once controlPlaneLoadBalancer is set | spec.controlPlaneLoadBalancer is write-once. See controlPlaneLoadBalancer fields. |
cannot be set after the cluster is created | The cluster was created without spec.controlPlaneLoadBalancer. Leaving the field out already means an external LoadBalancer. The endpoint is fixed when the cluster is created, so you cannot add the field later. |