Troubleshooting the global Cluster on Bare Metal

For the installation stages and the failures that all providers share, see Troubleshooting. The following table covers failures specific to Bare Metal. Run its commands on the bootstrap node. Unless a row says otherwise, the commands read the bootstrap cluster, which exists until the installation completes. If /var/cpaas/data/alauda.kubeconfig is missing, see Common Failures.

ResultAction
The first run does not print the ISO URLRun <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system get seedimage global-registration-iso -o yaml and read the SeedImageReady condition. If the registry does not serve tkestack/baremetal-base-image-iso:<isoImageTag>, push the images as described in OS Images in the Registry, then run create again. Otherwise, contact technical support.
The ISO build fails with SeedImageBuildFailureRead the cause with <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system logs global-registration-iso -c baseiso, fix it, and run create again. If it fails again, contact technical support.
A host does not appear as a MachineInventoryFrom the host console, check that the host booted the ISO and not an old OS from a disk. Check that it has a network address in the live ISO and can reach https://<bootstrapIP>:12443. An arm64 host must also provide SMBIOS. Fix any cause that you find. If all of these checks pass, contact technical support.
A MachineInventory shows status.install.phase=AwaitingTargetThe host is waiting for you to select its system disk. Select it as described in Register the Hosts.
A host stops before installing, and its MachineInventory lists a disk in status.install.staleSystemDisks with action: Blocked; the disk annotation is rejected; or a host returns to AwaitingTarget after its disk was selectedSee the matching row in Register Hosts: Troubleshooting. On the bootstrap node, run the commands with <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig in place of ac.
A MachineInventory is not Ready=True, or its status.plan.secretRef.name is emptyWait until the host has installed Alauda OS and rebooted, then list the inventories again with <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system get machineinventories.elemental.cattle.io -o yaml. If the inventory does not change, contact technical support.
The registration check fails on a poolCheck that the pool named in install-config.yaml exists in cpaas-system, sets spec.clusterName: global, and lists at least as many hosts as the replicas of the node groups that draw from it. Correct the pool, then run create again.
The registration check fails with pool members invalidThe MembersValid message names the host that does not exist or belongs to another pool. Correct the pool, then run create again.
Stage 4 does not advanceRun <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system describe baremetalmachine <name>. If it reports ImageCatalogMiss, see the next row. If the pool has no available MachineInventory, check that every host in the pool shows status.install.phase=Installed, or add a registered host to the pool. Otherwise, contact technical support.
A BaremetalMachine waits with ImageCatalogMissContact technical support to add the image catalog entry. The BaremetalMachine continues by itself once the entry exists.
Stage 7 waits for the agent handoverSave the global kubeconfig from the bootstrap cluster to a private file with (umask 077; <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system get secret global-kubeconfig -o jsonpath='{.data.value}' | base64 -d > <global-kubeconfig>). Then run <extract-dir>/res/<arch>/ac --kubeconfig <global-kubeconfig> -n cpaas-system get configmap baremetal-system-agent-handoff -o jsonpath='{.data.ready}'. While create runs, wait until this command prints true. If create stops at stage 7, run it again. If it stops again, contact technical support. Delete <global-kubeconfig> afterwards.
Stage 8 waits for the image catalogWith the same <global-kubeconfig>, run <extract-dir>/res/<arch>/ac --kubeconfig <global-kubeconfig> -n cpaas-system get configmap elemental-image-catalog -o yaml, and check that it has an entry for the Kubernetes version of the cluster. If create stops at stage 8, run it again. If it stops again, contact technical support.
curl … /healthz through the Self-built VIP does not print okCheck that the network allows VRRP (IP protocol 112, multicast 224.0.0.18) and gratuitous ARP between the control plane nodes. Also check that no other VRRP instance in the Layer 2 domain uses the same vrid. If both checks pass, contact technical support.

For provider conditions that also apply after installation, see Troubleshooting Bare Metal Workload Clusters.

The ISO Download URL Has Expired

The ISO download URL expires 120 minutes after the ISO is built. Hosts that have already booted the ISO continue to register. To get a new URL for the remaining hosts, run the same create command again on the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global create --dir <install-dir> --provider baremetal --reuse-manifests

create finds the expired ISO, builds it again, and prints the new download and checksum URLs. Boot the remaining hosts from the new ISO.