Local Kubernetes Clusters: A Comparison for Local Development and CI

Kubernetes, the powerful open-source container orchestration platform, has transformed the way we deploy and manage applications at scale. While Kubernetes is renowned for its effectiveness in distributed environments, it’s often overlooked that it can also be harnessed locally or in CI/CD workflows for development and testing purposes. In light of this, I delved into the world of running local Kubernetes clusters, putting seven different solutions to the test. This blog post complements my YouTube video on the topic, offering a concise summary of the key insights. For a more comprehensive exploration of each solution and a closer look at their application in common development scenarios, I encourage you to watch the accompanying video. Let’s dive in!

Solution Overview

The solutions presented in this blog post can be categorized into two groups: Standalone solutions that provide their own virtual machine (VM), and container-based solutions that run on a container engine. You may wonder why a VM is necessary in this context. The reason lies in the foundation of container technology, which relies on Linux kernel features. Therefore, to run containers, including containers on Kubernetes, a Linux environment is essential. As a result, on non-Linux systems, a VM is employed to create the required Linux environment and enable container execution.

(The later section about Minikube will explain why it is listed twice)

Evaluation Procedure

Each solution underwent comprehensive testing with the goal to assess its ability to deliver a local Kubernetes cluster with a feature set comparable to that of cloud providers. This involved evaluating the presence of a user-friendly dashboard for visualizing deployed resources, the availability of a metrics server for testing horizontal pod autoscalers, and the inclusion of a Kubernetes version selector. Emphasis was also placed on evaluating the ease-of-use and startup performance of each solution, as these factors greatly influence the overall experience and efficiency of running Kubernetes locally. The benchmark procedure and results are presented in a dedicated section.

Evaluation Results

Docker Desktop (Recommendation for local development):

Docker Desktop is a beginner-friendly Desktop app with a rock-solid user experience. In fact, it is so easy to use that many users won’t even notice that it has the above-mentioned VM under the hood. Kubernetes can be enabled via just one click in the settings. What’s particularly handy is that images built with the Docker CLI can be immediately executed in Kubernetes. While Docker Desktop excels in many aspects, it does have a few limitations. Its proprietary license model may be a drawback for some users, and it lacks certain features such as a built-in metrics server, Kubernetes dashboard, and the ability to select the Kubernetes version. Nonetheless, its maturity and extensive track record since 2016 contribute to its overall strength as a reliable solution for local Kubernetes clusters.

https://www.docker.com/products/docker-desktop/

Rancher Desktop (Recommendation for local development):

Rancher Desktop is a free and open source alternative to Docker Desktop, offering a seamless experience for beginners while placing a stronger emphasis on Kubernetes. It supports the use of the Docker CLI as well and thus matches the user experience of Docker Desktop. Additionally, Rancher Desktop stands out by offering a built-in Kubernetes dashboard, a Kubernetes version selector, and a built-in metrics server. Its user interface is nearly as polished and intuitive as Docker Desktop’s, making it a solid and enjoyable choice for local Kubernetes clusters.

https://rancherdesktop.io

Podman Desktop:

Podman Desktop is an open-source Desktop app that provides an alternative to Docker and Rancher Desktop. Unlike its counterparts, Podman Desktop doesn’t have Kubernetes directly built in. Instead, it serves as a container engine that can run other container-based Kubernetes solutions like Kind, Minikube, and K3D. Notably, Podman Desktop has recently introduced the ability to create Kind clusters through its user interface. While the UI still exhibits some quirks, such as unclear progress indication, Podman Desktop’s rapid pace of development makes it a promising solution worth keeping an eye on.

https://podman-desktop.io

K3D (Recommendation for CI pipelines):

K3D is a container-based Kubernetes solution equipped with a user-friendly CLI. Notably, it comes pre-packaged with a metrics server, supports multi-node clusters, and includes a Kubernetes version selector. Its standout feature, however, lies in its remarkable startup performance, making it an excellent choice for running integration tests within CI pipelines. While the project is primarily maintained by a single individual, it has been actively developed for over four years. During our attendance at KubeCon Europe this year, I had the pleasure of meeting Thorsten Klein, the project’s maintainer. K3D utilizes the K3S Kubernetes distribution under the hood, same as Rancher Desktop.

https://k3d.io

Kind:

As a container-based Kubernetes solution with a CLI, Kind offers a lightweight and user-friendly approach to running local Kubernetes clusters. It is a Kubernetes special interest group project that supports multi-node clusters and provides a Kubernetes version selector. Although its startup performance may not match that of K3D, Kind is a reliable choice for container-based local Kubernetes deployments.

https://kind.sigs.k8s.io

Minikube:

Minikube is the final of the three container-based solutions presented in this blog post.
What makes Minikube special is that it’s not just a container-based solution but supports also various VM technologies, making it a standalone solution as well. It is primarily a CLI, but since only a few weeks ago, a user interface is also available. Minikube had its initial release in 2016, same as Docker Desktop. This makes it one of the oldest projects on this list. Its feature set is very rich: It has a Kubernetes version selector, offers a metrics server, Kubernetes dashboard, and various other addons. However, in terms of startup performance, Minikube’s speed may not match that of K3D and Kind. Nevertheless, its flexibility and wide range of features make it a dependable choice for local Kubernetes deployments.

https://minikube.sigs.k8s.io

Microk8s:

Microk8s, a project by Canonical, renowned for their Ubuntu operating system, offers a CLI for running local Kubernetes clusters. It boasts essential features such as Kubernetes version selection, a built-in metrics server, and a user-friendly dashboard. However, Microk8s does have a notable weakness: its startup performance is the slowest among all the tools discussed in this post. This can be attributed to the fact that it redownloads several packages from the internet with each launch, resulting in increased initialization time. Despite this drawback, Microk8s remains a viable choice for users who prioritize its feature set, particularly those who use Microk8s as the Kubernetes distribution for their production environment.

https://microk8s.io

All solutions are available for macOS, Windows and Linux.

Comparison Table

To summarize the key aspects of each solution, the following comprehensive comparison table presents an overview of the features, strengths, and weaknesses of the seven solutions discussed in this post. This table can assist in selecting the ideal option for your local Kubernetes cluster setup.

Benchmarks

Startup performance

To benchmark the startup performance, the following procedure was conducted: Resetting the cluster, reinstalling a basic workload, and waiting for it to respond to traffic.

The performance comparison reveals that container-based solutions run fastest on Docker Desktop. In particular, K3D stands out as exceptionally fast. On the other hand, MicroK8s exhibits slower performance compared to the other solutions due to the need to download significant content with each launch.

Benchmarking notes: Each solution underwent six iterations, excluding the first one, to account for caching and to minimize the impact of network speed fluctuations on the results. Kind, K3D, and Minikube were additionally configured with registry mirrors to avoid redownloading container images. Docker Desktop, Rancher Desktop, Podman, and MicroK8s were each configured with 5 CPU cores and 8GB of memory. The tests were conducted on a MacBook Pro with the M1 Pro Apple Silicon Processor. To optimize K3D’s performance, unnecessary components were disabled. The benchmark code can be found on GitHub.

CI Pipeline performance

The following graph shows the performance in GitHub Actions CI pipelines:

Benchmarking notes: Each solution underwent twenty pipeline runs executed on GitHub Actions. To optimize K3D’s performance, unnecessary components were disabled. The benchmark code can be found on GitHub.

Conclusion

In conclusion, based on the extensive evaluation and benchmarking of the seven local Kubernetes cluster solutions, Rancher Desktop and Docker Desktop emerge as the top recommendations for local development due to their user-friendly interfaces and solid performance. Compared to Rancher Desktop, Docker Desktop has a more restrictive license but a slight lead in regard to user interface polish. For CI pipelines, K3D stands out as a robust choice, while Kind serves as a viable alternative. Throughout the assessment, the strengths and weaknesses of each tool were carefully examined, providing valuable insights for users seeking the most suitable solution for their specific requirements. While this blog post offers a concise summary, I highly recommend watching the accompanying YouTube video for a more comprehensive exploration of these tools in various development scenarios. The video provides a visual demonstration and further insights into the capabilities of each solution.

I would like to extend a special thanks to my colleagues Christoph Laufs and Bastian Hafer for their invaluable feedback on the draft YouTube video. Also, a special thanks goes to Max Rigling for editing and proofreading this blog post.