Troubleshoot a Workload Cluster Stuck in Provisioned

Use this guide when a workload cluster on Alauda OS has reached Cluster.status.phase = Provisioned and its control plane reports ready, but its nodes stay NotReady because the CNI is not deployed. It applies to every provider.

On Huawei Cloud Stack, if the KubeadmControlPlane does not become ready and a pool hostname contains a dot, this page does not apply; see Creation Stages.

Symptoms

Where to lookWhat you see
Cluster.status.phase on the global clusterProvisioned
KubeadmControlPlane.status.readytrue
Machine.status.phaseRunning
Machine condition NodeHealthystatus: False, reason: NodeConditionsFailed
MachineDeployment condition Availablestatus: False, reason: NotAvailable
Nodes of the workload clusterOne or more nodes are NotReady
Logs of the infrastructure provider on the global clusterRepeated ServiceAccount "sentry" not found errors while it reconciles the CNI AppRelease and the infrastructure cluster object

Diagnostic Steps

In the commands, <namespace> is the namespace of the cluster objects; for a cluster created with ac, it is cpaas-system.

Step 1 — Check the Cluster API Objects

ac config use-cluster global
ac get clusters.cluster.x-k8s.io <cluster-name> -n <namespace> -o wide
ac get kubeadmcontrolplane -n <namespace>
ac get machinedeployment -n <namespace>
ResultAction
The cluster is Provisioned and the KubeadmControlPlane is readyContinue with Step 2.
OtherwiseThis page does not apply. See the Troubleshooting page of your provider.

Step 2 — Check the Machine Conditions

ac config use-cluster global
ac get machine -n <namespace> \
  -l cluster.x-k8s.io/cluster-name=<cluster-name> \
  -o jsonpath='{range .items[*]}{.metadata.name}{" phase="}{.status.phase}{" "}{range .status.conditions[*]}{.type}={.status} {end}{"\n"}{end}'

Continue with Step 3 when a machine shows phase=Running with NodeHealthy=False; otherwise, this page does not apply.

Step 3 — Check the Provider Logs

ac config use-cluster global
ac logs -n cpaas-system <infra-provider-manager-pod> --tail=200 | \
  grep -E 'ServiceAccount|sentry|<cluster-name>'

Continue with Step 4 when the logs repeat ServiceAccount "sentry" not found. Otherwise, this page does not apply; on Huawei Cloud Stack, see Kube-OVN Is Not Installed.

Step 4 — Check the Import of the Cluster

Run each check. A healthy cluster passes all four.

  1. The platform cluster record exists:

    ac config use-cluster global
    ac get clusters.platform.tkestack.io <cluster-name>

    Expected: the record is listed, not NotFound.

  2. The Cluster API Cluster carries the capi.cpaas.io/imported label:

    ac get clusters.cluster.x-k8s.io <cluster-name> -n <namespace> -o json | \
      jq -r 'if (.metadata.labels | has("capi.cpaas.io/imported"))
             then "present (value=\"\(.metadata.labels["capi.cpaas.io/imported"])\")"
             else "missing"
             end'

    Expected: present (value="").

  3. Exactly one ClusterCredential is labelled for the cluster:

    ac get clustercredentials.platform.tkestack.io -o json | \
      jq -r --arg c <cluster-name> \
        '.items[] | select(.metadata.labels["cluster.x-k8s.io/cluster-name"]==$c) | "\(.metadata.name) \(.metadata.creationTimestamp)"'

    Expected: one line.

  4. The sentry ServiceAccount exists in the workload cluster:

    ac config sync
    ac config use-cluster <cluster-name>
    ac -n cpaas-system get serviceaccount sentry
    ac config use-cluster global

    Expected: the ServiceAccount is listed.

    If ac config use-cluster <cluster-name> fails, read the ServiceAccount with the admin kubeconfig of the cluster instead. Keep the file private (chmod 600 <cluster-name>.kubeconfig); it holds administrator credentials of the cluster.

    ac config use-cluster global
    ac -n <namespace> get secret <cluster-name>-kubeconfig \
      -o jsonpath='{.data.value}' | base64 -d > <cluster-name>.kubeconfig
    chmod 600 <cluster-name>.kubeconfig
    ac --kubeconfig <cluster-name>.kubeconfig -n cpaas-system get serviceaccount sentry
    rm -f <cluster-name>.kubeconfig
ResultAction
All four checks passThe import is complete. See the Troubleshooting page of your provider.
Any check failsContact technical support. Provide the cluster name and namespace, the output of Steps 1 to 4, and the names and creation timestamps of the ClusterCredential objects from check 3.
WARNING

Do not delete and recreate the cluster on your own: deleting it deletes its virtual machines on the IaaS platform. Restarting the cluster-transformer Pod on the global cluster does not import the cluster again.

See Also