Create a Cluster on Huawei DCS

ACP CLI (ac) renders the Cluster API and DCS objects from one install-config.yaml, applies them to the global cluster, and reports the creation stages. Every object lives in the cpaas-system namespace of the global cluster.

Prerequisites

  • You meet the Huawei DCS prerequisites.
  • ac is logged in to the platform as a platform administrator (ac login https://<platform-address> --name <session-name>).

Procedure

1. Write install-config.yaml

Create an installation directory for the cluster and write install-config.yaml in it. ac writes manifests/ and logs/ac.log next to it. From that file, ac renders one DCSIpHostnamePool and one DCSMachineTemplate for each node group. You therefore do not create IP pools or machine templates.

Copy the following template and fill in the placeholders. The table below the template describes the numbered lines, which are specific to Huawei DCS. For the other fields, see Common install-config.yaml Fields. For the objects that ac renders from this file, see Parameters.

install-config.yaml
cluster:
  clusterName: <cluster-name>
  sshKeys:
    - <ssh-public-key>
  # registrySecretRef:
  #   name: <registry-secret-name>
  #   namespace: cpaas-system
  # encryptionProviderConfig: <encryption-key>
  ntpServers: []
  network:
    podCIDR: 10.199.0.0/16
    serviceCIDR: 10.96.0.0/12
    kubeOVN:
      joinCIDR: 100.64.0.0/16
    loadBalancer:
      type: external
      address: <control-plane-endpoint>
      port: 6443
      # vrid: <vrid>
      # interface: <nic-name>

provider:
  dcs:
    vmTemplateName: <vm-template-name>
    controlPlaneHA: <control-plane-ha>
    machineFolder: <folder-name>
    # cdRomDatastoreName: <datastore>
    # osType: alaudaos-k
    # rootDisk:
    #   sizeGiB: <root-disk-gib>
    network:
      dvSwitchName: <dv-switch-name>
      portGroupName: <port-group-name>
    disks:
      datastore:
        type: <datastore-type>
        name: <datastore-name>
      isThin: false
      pciType: VIRTIO

controlPlane:
  replicas: 3
  cpuCore: 8
  memoryGiB: 16
  # dcs:
  #   machineFolder: <folder-name>
  #   cdRomDatastoreName: <datastore>
  #   rootDisk:
  #     sizeGiB: <root-disk-gib>
  network:
    gateway: <cp-gateway>
    # dns: [<dns-ip>]
    mask: <prefix-length>
    additions: []
    # additions:
    #   - mask: <prefix-length>
    #     dcs:
    #       dvSwitchName: <dv-switch-name>
    #       portGroupName: <port-group-name>
  disks:
    ephemeral:
      - name: var-lib-containerd
        sizeGiB: 100
      - name: var-lib-kubelet
        sizeGiB: 100
      - name: var-lib-etcd
        sizeGiB: 20
    persistent:
      - name: var-cpaas
        sizeGiB: 100
    rawDisks: [] 
    # rawDisks:
    #   - name: data-001
    #     sizeGiB: 50
    # dcs:
    #   datastore:
    #     type: <datastore-type>
    #     name: <datastore-name>
    #   isThin: true
    #   pciType: VIRTIO
  hosts:
    - hostname: <cp-0-hostname>
      ip: <cp-0-ip>
      additionalIps: []
    - hostname: <cp-1-hostname>
      ip: <cp-1-ip>
      additionalIps: []
    - hostname: <cp-2-hostname>
      ip: <cp-2-ip>
      additionalIps: []

computes:
  - name: worker
    replicas: 3
    cpuCore: 8
    memoryGiB: 16
    # dcs:
    #   machineFolder: <folder-name>
    #   cdRomDatastoreName: <datastore>
    #   rootDisk:
    #     sizeGiB: <root-disk-gib>
    network:
      gateway: <worker-gateway>
      # dns: [<dns-ip>]
      mask: <prefix-length>
      additions: []
      # additions:
      #   - mask: <prefix-length>
      #     dcs:
      #       dvSwitchName: <dv-switch-name>
      #       portGroupName: <port-group-name>
    disks:
      ephemeral:
        - name: var-lib-containerd
          sizeGiB: 100
        - name: var-lib-kubelet
          sizeGiB: 100
      persistent:
        - name: var-cpaas
          sizeGiB: 100
      rawDisks: []
      # rawDisks:
      #   - name: data-001
      #     sizeGiB: 50
      # dcs:
      #   datastore:
      #     type: <datastore-type>
      #     name: <datastore-name>
      #   isThin: true
      #   pciType: VIRTIO
    hosts:
      - hostname: <worker-0-hostname>
        ip: <worker-0-ip>
        additionalIps: []
      - hostname: <worker-1-hostname>
        ip: <worker-1-ip>
        additionalIps: []
      - hostname: <worker-2-hostname>
        ip: <worker-2-ip>
        additionalIps: []

credentialRef:
  name: <credential-secret-name>
  namespace: cpaas-system
#FieldDescription
cluster.networkRequired · objectThe cluster network.
1cluster.network.loadBalancerRequired · objectThe control plane endpoint. type is external (the default) for your own load balancer, or internal for a Self-built VIP that the provider runs on the control plane nodes. address is the load balancer address or the VIP: an IPv4 address that is not a node address and is outside the Pod, Service, and join CIDRs. port defaults to 6443. With internal, uncomment vrid, which is required. It is a VRRP router ID from 1 to 255 that is unique in the control plane Layer 2 domain. You can also uncomment interface, the NIC that holds the VIP, which by default is the NIC of the node IP. vipMode is not supported. See Control Plane Endpoint and HA.
provider.dcsRequired · objectThe Huawei DCS settings.
2provider.dcs.vmTemplateNameRequired · stringThe VM template of every node. It decides the Kubernetes version that ac installs; see VM Template and Kubernetes Version. Its Alauda OS version matches the platform version; see Version Mapping.
3provider.dcs.controlPlaneHARequired · booleantrue spreads the control plane VMs across DCS hosts with a DRS rule; see DCS Objects, Capacity, and Addresses for what it needs. false places them without that rule.
4provider.dcs.machineFolderRequired · stringThe name of an existing VM folder on DCS, in which the node VMs are created. Use a VM folder, not a cluster or storage folder. Enter the folder name only, not a path; it must match exactly. ac does not create the folder.
5provider.dcs.cdRomDatastoreNameOptional · string · Default: the datastore of the system diskThe datastore of the CD-ROM that carries each node's Ignition ISO.
6provider.dcs.osTypeOptional · string · Default: Alauda OS M55Set it to alaudaos-k for a template of Alauda OS KO or KU; see OS Variants. Otherwise, leave it out.
7provider.dcs.rootDiskOptional · object · Default: the VM template's system disk sizesizeGiB is the system disk size. DCS rejects a size smaller than the template's.
8provider.dcs.networkRequired · objectThe primary NIC of every node: dvSwitchName and a portGroupName on that switch.
9provider.dcs.disks.datastoreConditional · objectWhere every disk of the node is created. type is Datastore for a single datastore or DatastoreCluster for a datastore cluster, and name is its name. Required unless every node group sets disks.dcs.datastore. See Disks.
10provider.dcs.disks.isThinOptional · boolean · Default: falsetrue thin-provisions the persistent and raw disks; false thick-provisions them. The ephemeral disks are not affected.
11provider.dcs.disks.pciTypeOptional · string · Default: VIRTIOThe bus of the persistent and raw disks: VIRTIO, SCSI, or IDE.
controlPlaneRequired · objectThe control plane node group.
12controlPlane.cpuCore, controlPlane.memoryGiBOptional · integer · Default: 8 and 16The VM size of the node group: the number of CPU cores and the memory in GiB, each greater than 0.
13controlPlane.dcsOptional · object · Default: the values of provider.dcsFor this group, overrides machineFolder, cdRomDatastoreName, and rootDisk of provider.dcs, with the same rules. Uncomment the block and keep only the fields that differ.
14controlPlane.networkRequired · objectThe NICs of the node. gateway is the IPv4 gateway of the primary NIC. mask is its prefix length, 1 to 32. dns is optional, and DCS uses only the first entry. additions are the optional additional NICs, matched in order to hosts[].additionalIps. Use the commented block. In an addition, mask is 1 to 32 and defaults to network.mask. additions defaults to []. See Additional NICs.
15controlPlane.network.additions[].dcsOptional · object · Default: the values of provider.dcs.networkThe switch and port group of the additional NIC: dvSwitchName and portGroupName.
16controlPlane.disks.ephemeralRequired · []objectThe disks var-lib-containerd, var-lib-kubelet, and var-lib-etcd, which are recreated with the VM. Keep the names and change only sizeGiB.
17controlPlane.disks.persistentRequired · []objectThe disk var-cpaas, which is kept when the VM is replaced. Keep the name and change only sizeGiB.
18controlPlane.disks.rawDisksOptional · []object · Default: []Raw block volumes for TopoLVM. Each has a name of your choice and a sizeGiB greater than 0. Use the commented block. See Raw Disks for TopoLVM.
19controlPlane.disks.dcsOptional · object · Default: the values of provider.dcs.disksFor this group, overrides datastore, isThin, and pciType of provider.dcs.disks. The datastore override takes both of its fields. Uncomment the block and keep only the fields that differ.
20controlPlane.hostsRequired · []objectOne entry per node, and at least replicas entries, because DCS assigns no node identities. hostname is the node hostname. ip is the IPv4 address of the primary NIC. additionalIps lists one IPv4 address per network.additions entry, in order; the default is []. See Node Addresses and Placement.
computes[]Optional · []object · Default: []The worker node groups, one entry per group. The controlPlane fields above also apply to each entry. The rows below describe the numbered lines of the worker entry.
21computes[].dcsOptional · object · Default: the values of provider.dcsWorks as controlPlane.dcs does.
22computes[].disks.ephemeralRequired · []objectA worker node group has no var-lib-etcd. Keep var-lib-containerd and var-lib-kubelet, and change only sizeGiB.

2. Create the Cluster

WARNING

Before you run create, make sure that the password in the credential Secret is correct, because DCS locks the account after repeated failed sign-ins.

Run the following command from a machine where ac is logged in to the platform:

ac config use-cluster global
ac adm alpha cluster create --dir <install-dir> --provider dcs

ac first checks the credential Secret, the DCS objects named in install-config.yaml, the ConfigMap of the VM template, and the registry. If a check fails, nothing is changed and the error names the item.

create returns once the objects are applied, and the cluster keeps building. To wait until all stages are ready, add --wait. --timeout (default 2h) limits only the wait. To render without applying, run ac adm alpha cluster manifests with the same flags.

3. Monitor the Creation

ac config use-cluster global
ac adm alpha cluster status --cluster <cluster-name>                      # prints the four stages once
ac adm alpha cluster wait-for install-complete --cluster <cluster-name>   # waits until all four are ready

Continue when all four stages are ready; otherwise see Troubleshooting.

4. Verify the Cluster

Run the following commands from a machine where ac is logged in to the platform:

ac config sync
ac config use-cluster <cluster-name>
ac get nodes
ac config use-cluster global

Continue when every node is Ready; otherwise see Troubleshooting.

Self-built VIP

Applies to: cluster.network.loadBalancer.type: internal. Skip it for an external load balancer.

ac config use-cluster global
ac -n cpaas-system get dcscluster <cluster-name> \
  -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneEndpoint.host}{":"}{.spec.controlPlaneEndpoint.port}{"\n"}'
ac -n cpaas-system get moduleinfo \
  -l cpaas.io/cluster-name=<cluster-name>,cpaas.io/module-name=alive
ac config sync
ac config use-cluster <cluster-name>
ac -n cpaas-system get apprelease alive
ac -n kube-system get pods -l app=alive -o wide
ac config use-cluster global

Continue when the output shows internal with your VIP and port, an alive ModuleInfo, a synced and healthy alive AppRelease, and a Ready alive Pod on every control plane node; otherwise see Self-built VIP in Troubleshooting.

Then log in to a control plane node of the cluster as the boot user (ssh boot@<control-plane-node-ip>). Use the private key of one of the SSH public keys in cluster.sshKeys. Only root can read the cluster CA file, so run curl with sudo:

sudo curl --cacert /etc/kubernetes/pki/ca.crt https://<vip>:6443/healthz

Continue when it prints ok; otherwise see Self-built VIP in Troubleshooting.

Control Plane HA

Applies to: provider.dcs.controlPlaneHA: true. Skip it when it is false.

ac config use-cluster global
ac -n cpaas-system get dcscluster <cluster-name> \
  -o jsonpath='{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}'

Continue when the status is True; otherwise see Control Plane HA in Troubleshooting.

Additional NICs

Applies to: node groups with network.additions. Skip it otherwise.

ac config use-cluster global
ac -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=<cluster-name> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.networkConfig.ip}{"\t"}{.status.additionalNic}{"\n"}{end}'

Continue when each DCSMachine lists one additional NIC per address in its host's additionalIps; otherwise contact technical support.

Run create Again

create only creates the cluster. Do not run it on a cluster that already exists. To change a running cluster, see Manage Nodes.

If create stops before the cluster is created, run the same command again. Completed work is skipped. When you run create again on the same directory:

  • It recreates objects that were deleted. The exception is a Secret: ac creates Secrets once and then leaves them as they are.
  • It does not overwrite a field that you changed with another tool. When the rendered value differs, create reports that the fields are owned by another manager and stops. Do not delete the object that the error names.
  • When manifests/ already exists, create asks whether to reuse it. Press Enter to reuse it, or answer n to render again after you change install-config.yaml. When standard input is not a terminal, pass --reuse-manifests=true or --reuse-manifests=false instead.
  • Reusing manifests/ ignores install-config.yaml and skips every check.

Do not edit the files in manifests/.

WARNING

Keep <install-dir>/manifests/, or set cluster.encryptionProviderConfig. A generated encryption key is kept only in manifests/. If you render again without doing either, a new key is generated, and manifests/ then no longer records the key that the cluster uses.

Next Steps