Empowering Kubernetes capabilities with lightweight K8s distros

Small Wonders

Decision Supports

MicroK8s was initially meant for developer workstations and was only subsequently optimized further for a low-memory footprint. It favors easy extensibility over minimal resource usage and may be a preferred choice when using Ubuntu and the snap package manager, although it also can be installed in Windows. High availability in MicroK8s is activated automatically on clusters with three or more nodes. Patch release updates are installed automatically in MicroK8s. The tests confirmed the convenient activation of prebuilt add-ons (e.g., dashboard, Prometheus) with single commands, which in contrast typically requires a number of configuration steps for the other distributions.

K3s has been designed to reduce binary size (64.5MB) and memory footprint targeting resource-constrained environments such as edge clusters as well as developer workstations. Memory footprint was reduced by combining all components into a single process, and binary size was reduced by removing third-party storage drivers and cloud providers, for example, from K8s. Updates follow the upstream K8s releases, with a goal of patch release within one week and new minor releases within 30 days. K3s has found widespread popularity because of its light weight and easy install procedure, with only most essential components included, and it could be a preferred choice for resource-constrained devices, especially if those are dedicated worker nodes.

k0s appears similar to K3s and is also rigorously tuned for low resource usage and easy installation. It has so far only had low popularity. The developers highlight that the design was done with security in mind, allowing 100 percent FIPS compliance if a proper toolchain is in place. Because the k0s binary is self-contained, security vulnerabilities can potentially be fixed quickly, not being dependent on external components. k0s intentionally reduces bundled add-ons, such as ingresses, service meshes, and storage, because they follow opinionated design patterns, making them less flexible and harder to maintain in some scenarios. In these tests, k0s appears to be lean and streamlined, with a very easy install procedure.

Conclusions

The tests showed marginal differences in control plane performance between the lightweight K8s distributions, with MicroK8s having slightly poorer performance than K3s and k0s, which performed almost identically in pod throughput and pod creation latency. The performance was measured for the default configurations of the distributions, although default Metrics Servers were deactivated for all and k0s was reconfigured to allow combined controller worker nodes to achieve a fair comparison. Further performance optimization might be possible by reconfiguring the K8s distributions or integrating custom components through their different extension mechanisms.

Control plane tests were performed on artificially created extreme stress scenarios, which might be rare in day-to-day practice, especially on resource-constrained edge devices. For more modest, regular workloads, you might not perceive significant performance differences among the K8s distribution used. However, the results here confirmed that the creation of more than 100 pods is feasible on a single node for all distributions, which implies that the K8s distributions set no limits on possible workloads when constrained by the available hardware.

Data plane performance showed some more significant differences between the distributions, with K3s having poor performance on the combined controller/worker node in this scenario. This result seems to correlate with the high controller resource usage of K3s in idle conditions. Possibly the continuous resource usage of the control plane components decreased the achievable throughput in this scenario. However, in practice, such high-load scenarios might use dedicated worker nodes, which would be released from control plane resource usage. On some standalone edge devices with challenging machine learning workloads, the data plane performance could be relevant. However, when the lightweight K8s distributions are used on powerful developer workstation, it should be less of a problem. The results also indicated that the data plane performance is very similar between MicroK8s and K3s; however, they did not fully exhaust the available resources.

Get your hands dirty by trying out each distribution (Table 2). To know more about the Kubernetes distributions and their importance in the cloud-native landscape, head on over to the official Kubernetes documentation [6].

Table 2

Performance and Architecture Comparison

Feature or Aspect	k0s	K3s	MicroK8s
Control plane architecture	Bare-bones control plane, only core services (API server, scheduler, controller manager). No workloads allowed on control nodes.	Lightweight but includes additional system services like `containerd`. Workloads can be scheduled on control nodes unless restricted by `taints` or `tolerations`.	Modular traditional control plane. Control plane and workloads can run together. Flexible optimization for different environments.
Management processes	Runs as a simple Linux process, with no unnecessary services. Highly efficient in terms of resources.	More system services are bundled by default, leading to a slightly larger footprint than k0s.	Uses containerized components; flexible for different environments and scales well in edge or cloud environments.
Workload on the control plane	Not allowed. Strict separation of control plane and workloads for enhanced reliability.	Allowed unless specifically restricted by `taints` or `tolerations`, meaning less strict separation of workloads and management.	Can run workloads on control nodes, which is especially useful in single-node setups. Makes it more flexible but can consume resources meant for control plane management.
Minimum CPU cores required	1	1	2
Minimum RAM required (GB)	1	1	4
Total memory consumption (MB)	510	512	540
Security	Secure by design with minimized attack surface. Only one port is required for communication between control and worker nodes, which simplifies firewall rules and reduces potential vulnerabilities.	Security depends on the configuration. Requires opening multiple ports for different services, increasing potential attack surface if not properly secured.	Offers flexible security depending on the setup, but a larger set of services and ports might be open depending on enabled components (e.g., network plugins).
Datastore uses	SQLite as the default datastore, which can be scaled to etcd for high-availability setups. Keeps the data layer simple in low-resource environments.	SQLite as the default datastore with an option to use etcd for high availability, but allows workloads on the control plane.	By default, can use Dqlite, which is a Raft-based lightweight database, or etcd when high availability is required. Like K3s, workloads can run on the control plane.
Networking (CNI)	Simple network configuration with one port required for node communication, reducing the need for complex firewall rules.	Slightly more complex networking, requiring more than one port for various services depending on the setup.	Offers multiple container network interfaces (CNIs; Calico, Flannel), with more flexibility in networking, but requires configuring ports on the basis of the selected CNI solution.
CNI interface	k0s doesn't bundle in the default CNI plugin, giving you more flexibility in choosing your networking setup. Users can add Calico, Cilium, or others as needed.	Uses Flannel as the default network back end, which is sufficient for most lightweight deployments but might lack the advanced features found in more complex network solutions like Calico.	Provides built-in support for Calico and Flannel as networking solutions. Calico is more feature rich and supports network policies, whereas Flannel is lighter and easier to configure for simpler use cases.
Container runtime	Delegates runtime management to the workers, focusing solely on control components. Uses a connectivity agent for communication.	Bundled `containerd` runtime by default, which increases the size and footprint of the control plane.	Uses `containerd` as the default runtime, with the ability to enable or disable depending on the setup. Containerized control plane components increase flexibility but could increase complexity.
Deployment use cases	Best suited for environments where strict separation between control and workloads is needed and resource efficiency is critical (e.g., edge, IoT, or secure deployments).	Suited for resource-constrained environments where some flexibility between control and workload nodes is acceptable. Great for edge devices and small systems like Raspberry Pi.	Ideal for single-node, multinode, and cloud setups with flexible component selection and scaling options. Can be used in both small and production-grade clusters.

Infos

k0s: https://k0sproject.io/
K3s: https://k3s.io/
MicroK8s: https://microk8s.io/
k0s architecture: https://docs.k0sproject.io/v1.23.5+k0s.0/architecture/?h=architecture
Koziolek, H., and N. Eskandani. "Lightweight Kubernetes distributions: A performance comparison of MicroK8s, K3s, k0s, and MicroShift." In: ICPE'23. Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (Association for Computing Machinery, 2023), pg. 17-29, https://dl.acm.org/doi/10.1145/3578244.3583737
Kubernetes distro partners: https://kubernetes.io/docs/home/

The Author

Aryan Parashar is Developer Advocate and Ambassador at Cloud Native Computing Foundation (CNCF) landscape projects and creates YouTube videos about cloud-native WebAssembly (Wasm), sustainability, machine learning (ML)Ops, large language model (LLM)Ops, and open source. You can find his content at https://www.youtube.com/@AryanParashar_.

« Previous 1 2 3 4 5