Disk Configuration

Storage Capacity

The disks of a node depend on the environment and on the role of the node. In this section, IaaS means Alauda OS on Huawei DCS, Huawei Cloud Stack, or VMware vSphere.

Disk Layout by Environment

IaaS

Each of the following paths needs its own disk:

  • The system disk.
  • A disk for /var/lib/etcd, on control plane nodes only.
  • A disk for /var/lib/containerd.
  • A disk for /var/lib/kubelet.
  • A disk for /var/cpaas, declared as a persistent disk.

The provider keeps the persistent /var/cpaas disk when it replaces the node. Declare the other disks as standard disks.

Bare Metal on Alauda OS

Use one system disk for all paths. The provider does not wipe the host disk when it upgrades or updates the host. Size the system disk by node role; see Disks by Node Role.

Bare Metal on a Non-Alauda OS

Use one system disk, as on Alauda OS, with /cpaas in place of /var/cpaas. Create the paths as LVM logical volumes so that they can be expanded later.

A node that runs TopoLVM also needs a raw disk in every environment; see Raw Disk for TopoLVM. On Alauda OS, declare the disks in the machine configuration of the provider; see Create Workload Clusters on Alauda OS.

Path Sizes

PathUsed forSize
/Operating systemAt least 50 GB; 100 GB recommended.
/var/lib/etcdetcd data, on control plane nodes onlyAt least 10 GB; 20 GB recommended.
/var/lib/containerdContainer images and container writable layersAt least 100 GB; 150 GB recommended.
/var/lib/kubeletPod ephemeral data, such as emptyDir volumesAt least 100 GB; 200 GB recommended on global cluster nodes and 100 GB on workload cluster nodes.
/var/cpaas on Alauda OS
/cpaas on a non-Alauda OS
Platform data: etcd local backups, the log collection buffer, and audit dataAt least 100 GB on control plane nodes and 50 GB on other nodes; 200 GB recommended on log storage nodes.
  • /: keep utilization below 80%. Above this threshold, Pods on the node may be evicted.
  • /var/lib/etcd: use a dedicated high-IO disk; see Recommended ETCD Practices.
  • /var/lib/kubelet: the nodes of a Single Cluster count as global cluster nodes.
  • /var/cpaas: log storage nodes are the nodes that run ClickHouse and Kafka. On other nodes, the minimum size is also the recommended size.
  • /var/cpaas on control plane nodes holds etcd local backups in /var/cpaas/backup; see etcd Backup and Restore.

Raw Disk for TopoLVM

A node that runs TopoLVM needs an additional raw disk, on IaaS and on Bare Metal alike. TopoLVM manages this disk, so do not format or mount it.

Workloads on the nodeIaaSBare Metal
Logging, monitoring, and the registry750 GB1 TB
Logging and monitoring650 GB1 TB
Logging only600 GB1 TB
Monitoring only100 GB512 GB
  • On IaaS, size the raw disk for the current need. The disk can be expanded later when more capacity is needed.
  • On Bare Metal, raw disks usually come in sizes such as 512 GB and 1 TB. Select the smallest size that covers the need.
  • The logging sizes cover about 5,000 log entries per second with 7 days of retention. For more logs, add capacity to the TopoLVM volume group.
  • The registry is the Registry with 1 replica, which runs on the infra nodes of a Single Cluster.
  • The infra nodes of a DevOps cluster run monitoring and logging, like workload infra nodes, and use the sizes of the logging and monitoring row.

In the DevOps planning, Harbor, GitLab, SonarQube, Nexus, and their PostgreSQL and Redis instances keep their data in Ceph (RBD and RGW). They therefore need no TopoLVM disk. For the OSD disks of the storage nodes, see Harbor on the global Cluster and Dedicated DevOps Cluster.

Disks by Node Role

The following table summarizes the recommended disks of common node roles. The nodes of a Single Cluster follow the global cluster columns.

Diskglobal control plane nodeglobal infra nodeWorkload control plane nodeWorkload infra or log nodeApplication worker node
IaaS
System disk100 GB100 GB100 GB100 GB100 GB
/var/lib/etcd20 GB—20 GB——
/var/lib/containerd150 GB150 GB150 GB150 GB150 GB
/var/lib/kubelet200 GB200 GB100 GB100 GB100 GB
/var/cpaas, persistent100 GB200 GB100 GB200 GB50 GB
Raw disk for TopoLVM—650 GB—650 GB on infra nodes; 600 GB on log nodes—
Bare Metal
System disk, minimum512 GB512 GB512 GB512 GB512 GB
System disk, recommended1 TB1 TB512 GB1 TB512 GB
Raw disk for TopoLVM—1 TB—1 TB—
  • Application worker nodes run the business workloads. The table lists only the platform disks that they need.
  • On IaaS, the infra nodes of a Single Cluster also run the registry and need a 750 GB raw disk for TopoLVM.
  • Monitor, gateway, and storage nodes use the disks of the application worker node column. Monitor nodes also need a raw disk for TopoLVM: 100 GB on IaaS and 512 GB on Bare Metal.
  • For the disks of storage nodes for Ceph, see Planning Your Deployment.
  • On Bare Metal, each system disk size covers the sum of the recommended sizes of the paths on that node.

Fast storage is essential for etcd to perform reliably. etcd depends on durable, low-latency disk operations to persist proposals to its write-ahead log (WAL).
If disk writes take too long, fsync delays can cause the member to miss heartbeats, fail to commit proposals promptly, and experience request timeouts or temporary leader changes. These issues can also slow the Kubernetes API and degrade overall cluster responsiveness.
In conclusion, HDDs are a poor choice and are not recommended. If you must use HDDs for etcd, choose the fastest available (for example, 15,000 RPM).

INFO

The following hard drive practices provide optimal etcd performance:

  • Prefer SSDs or NVMe as etcd drives. When write endurance and stability are priorities, consider server-grade single-level cell (SLC) SSDs. Avoid NAS, SAN, and HDDs.

    • Prefer drives with high write throughput to accelerate compaction and defragmentation.
    • Prefer drives with strong read bandwidth to reduce recovery time after failures.
    • Prefer drives with consistently low latency to ensure fast read and write operations.
  • Avoid distributed block storage systems such as Ceph RADOS Block Device (RBD), Network File System (NFS), and other network-attached backends, because they introduce unpredictable latency.

  • Keep etcd data on a dedicated drive or a dedicated logical volume.

    • Do not place I/O-sensitive (such as logging) or other intensive filesystem activity on control-plane hosts, or at least do not let them share the same underlying storage with etcd.
  • Continuously benchmark with tools like fio and use the results to track performance as the cluster grows. For more information, see the disk benchmarking guide.

Validating the hardware for etcd

SpecificationMinimum RequirementRecommendedNotes
Sequential write IOPS50500 (higher is better)Most cloud providers publish concurrent IOPS rather than sequential IOPS. The concurrent IOPS values are typically about 10× higher than sequential ones.
Disk bandwidth10 MB/s100 MB/s (higher is better)Higher disk bandwidth allows faster data recovery when a failed member needs to catch up with the cluster.
Throughput (sequential 8 kB write with fdatasync)50 writes per 10 ms500 writes per 2 msReflects sustained write throughput when data is flushed to disk after each write operation.

Benchmarking with fio

To measure actual sequential IOPS and throughput, use the disk benchmarking tool fio:

WARNING

Do not run these tests against any nodes of the clusters.
Instead, run the tests against a dedicated VM that has the same set up as the control plane nodes.

set -e
mkdir -p /var/lib/etcd/

echo "INFO: Running fio"
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf --output-format=json --runtime=60 --time_based=1 | tee /tmp/fio.out

# Scrape the fio output for p99 of fsync in ns
fsync=$(cat /tmp/fio.out | jq '.jobs[0].sync.lat_ns.percentile["99.000000"]')
iops=$(cat /tmp/fio.out | jq '.jobs[0].write.iops')
echo "INFO: 99th percentile of fsync is $fsync ns"

# Compare against the recommended value
if [[ $fsync -ge 10000000 ]]; then
    echo "WARN: IOPS is $iops, 99th percentile of the fsync is greater than the recommended value which is ${fsync} ns > 10 ms, faster disks are recommended to host etcd for better performance"
else
    echo "INFO: IOPS is $iops, 99th percentile of the fsync is within the recommended threshold: - 10 ms, the disk can be used to host etcd"
fi