Disk Configuration
TOC
Storage CapacityDisk Layout by EnvironmentIaaSBare Metal on Alauda OSBare Metal on a Non-Alauda OSPath SizesRaw Disk for TopoLVMDisks by Node RoleRecommended ETCD PracticesValidating the hardware for etcdBenchmarking with fioStorage Capacity
The disks of a node depend on the environment and on the role of the node. In this section, IaaS means Alauda OS on Huawei DCS, Huawei Cloud Stack, or VMware vSphere.
Disk Layout by Environment
IaaS
Each of the following paths needs its own disk:
- The system disk.
- A disk for
/var/lib/etcd, on control plane nodes only. - A disk for
/var/lib/containerd. - A disk for
/var/lib/kubelet. - A disk for
/var/cpaas, declared as a persistent disk.
The provider keeps the persistent /var/cpaas disk when it replaces the node. Declare the other disks as standard disks.
Bare Metal on Alauda OS
Use one system disk for all paths. The provider does not wipe the host disk when it upgrades or updates the host. Size the system disk by node role; see Disks by Node Role.
Bare Metal on a Non-Alauda OS
Use one system disk, as on Alauda OS, with /cpaas in place of /var/cpaas. Create the paths as LVM logical volumes so that they can be expanded later.
A node that runs TopoLVM also needs a raw disk in every environment; see Raw Disk for TopoLVM. On Alauda OS, declare the disks in the machine configuration of the provider; see Create Workload Clusters on Alauda OS.
Path Sizes
/: keep utilization below 80%. Above this threshold, Pods on the node may be evicted./var/lib/etcd: use a dedicated high-IO disk; see Recommended ETCD Practices./var/lib/kubelet: the nodes of a Single Cluster count asglobalcluster nodes./var/cpaas: log storage nodes are the nodes that run ClickHouse and Kafka. On other nodes, the minimum size is also the recommended size./var/cpaason control plane nodes holds etcd local backups in/var/cpaas/backup; see etcd Backup and Restore.
Raw Disk for TopoLVM
A node that runs TopoLVM needs an additional raw disk, on IaaS and on Bare Metal alike. TopoLVM manages this disk, so do not format or mount it.
- On IaaS, size the raw disk for the current need. The disk can be expanded later when more capacity is needed.
- On Bare Metal, raw disks usually come in sizes such as 512 GB and 1 TB. Select the smallest size that covers the need.
- The logging sizes cover about 5,000 log entries per second with 7 days of retention. For more logs, add capacity to the TopoLVM volume group.
- The registry is the Registry with 1 replica, which runs on the infra nodes of a Single Cluster.
- The infra nodes of a DevOps cluster run monitoring and logging, like workload infra nodes, and use the sizes of the logging and monitoring row.
In the DevOps planning, Harbor, GitLab, SonarQube, Nexus, and their PostgreSQL and Redis instances keep their data in Ceph (RBD and RGW). They therefore need no TopoLVM disk. For the OSD disks of the storage nodes, see Harbor on the global Cluster and Dedicated DevOps Cluster.
Disks by Node Role
The following table summarizes the recommended disks of common node roles. The nodes of a Single Cluster follow the global cluster columns.
- Application worker nodes run the business workloads. The table lists only the platform disks that they need.
- On IaaS, the infra nodes of a Single Cluster also run the registry and need a 750 GB raw disk for TopoLVM.
- Monitor, gateway, and storage nodes use the disks of the application worker node column. Monitor nodes also need a raw disk for TopoLVM: 100 GB on IaaS and 512 GB on Bare Metal.
- For the disks of storage nodes for Ceph, see Planning Your Deployment.
- On Bare Metal, each system disk size covers the sum of the recommended sizes of the paths on that node.
Recommended ETCD Practices
Fast storage is essential for etcd to perform reliably. etcd depends on durable, low-latency disk operations to persist proposals to its write-ahead log (WAL).
If disk writes take too long, fsync delays can cause the member to miss heartbeats, fail to commit proposals promptly, and experience request timeouts or temporary leader changes. These issues can also slow the Kubernetes API and degrade overall cluster responsiveness.
In conclusion, HDDs are a poor choice and are not recommended. If you must use HDDs for etcd, choose the fastest available (for example, 15,000 RPM).
The following hard drive practices provide optimal etcd performance:
-
Prefer SSDs or NVMe as etcd drives. When write endurance and stability are priorities, consider server-grade single-level cell (SLC) SSDs. Avoid NAS, SAN, and HDDs.
- Prefer drives with high write throughput to accelerate compaction and defragmentation.
- Prefer drives with strong read bandwidth to reduce recovery time after failures.
- Prefer drives with consistently low latency to ensure fast read and write operations.
-
Avoid distributed block storage systems such as Ceph RADOS Block Device (RBD), Network File System (NFS), and other network-attached backends, because they introduce unpredictable latency.
-
Keep etcd data on a dedicated drive or a dedicated logical volume.
- Do not place I/O-sensitive (such as logging) or other intensive filesystem activity on control-plane hosts, or at least do not let them share the same underlying storage with etcd.
-
Continuously benchmark with tools like
fioand use the results to track performance as the cluster grows. For more information, see the disk benchmarking guide.
Validating the hardware for etcd
Benchmarking with fio
To measure actual sequential IOPS and throughput, use the disk benchmarking tool fio:
Do not run these tests against any nodes of the clusters.
Instead, run the tests against a dedicated VM that has the same set up as the control plane nodes.