Of all the problems we have confronted, the ones over which the most brainpower, ink, and code have been spilled are related to managing configurations. - Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes, Google Inc.
I spent some time tracking down tools that seem like they manage K8S manifests. I found…many. This list of ~20 only touches the surface of a bubbling ecosystem.
Some of these tools are large and some are small. Some are simple templating systems, others can manage the entire life cycle of applications across many kinds of infrastructure. Some are libraries for programming languages, others complete languages themselves. A few are mature projects, some brand new, and others side projects of a single developer. The breadth and variance is quite amazing!
Please note that these are in no specific order, and neither does that order represent any kind of preference. Further, some of these tools, eg. Ksonnet, are no longer maintained. This is not an exhaustive list!
But let’s get to it.
Jsonnet - A data templating language. Used by many tools in this list behind the scenes. Powerful, but sometimes with great power comes with great, uh, cognitive load?
jk - “jk generates all your JSON, YAML and arbitrary text configuration files. With a little luck, you will not have to touch a YAML file again. Ever.” Bold claim!
Cue - Apparently created by the person (or persons) behind the Borg Configuration Language (BCL) used at Google. “CUE is an open source language, with a rich set APIs and tooling, for defining, generating, and validating all kinds of data: configuration, APIs, database schemas, code, … you name it.”
Isopod - “Isopod is an expressive DSL framework for Kubernetes configuration. Without intermediate YAML artifacts, Isopod renders Kubernetes objects as Protocol Buffers, so they are strongly typed and consumed directly by the Kubernetes API.” Interesting that they skip the YAML. Based on Skycfg. A good intro here.
Configula - “Configula is inspired by the JSX language in React that combines Javascript and HTML tags. Configula defines a similar pattern for Python and YAML (for now).” Joe Beda went over it in TKIG. It’s a Go utility that calls out to Python!
Ko - Point your K8S container image to a Go repo. It will build a container image using the binary generated from the repo. Does not use Docker to actually build the image. Wraps kubectl in some places.
Kustomize - “kustomize targets kubernetes; it understands and can patch kubernetes style API objects. It’s like make, in that what it does is declared in a file, and it’s like sed, in that it emits edited text.” I’ve seen Kustomize used a few times; seems popular. Also it is conveniently part of Kubectl.
Helm - Helm is likely the most commonly used tool on this list. Recently Helm 3 was released removing its main criticism regarding the use of the tiller server.
Kubecfg - “Kubecfg relies heavily on jsonnet to describe Kubernetes resources, and is really just a thin Kubernetes-specific wrapper around jsonnet evaluation.” ‘nuff said!
Ksonnet - Ksonnet is no longer worked on, but I think it’s a good thing to have in this list as a reminder ensure that not every tool will stick around. Experimentation will continue in this space.
k14s - “We believe that working with simple, single-purpose tools that easily interoperate with one another results in a better, workflow compared to the all-in-one approach chosen by Helm. We have found this approach to be easier to understand and debug.” k14s is developed by people at Pivotal (where I work, now part of VMware). There’s a TGIK episode on it. The Unix philosophy has done us all well over time, perhaps this is the right approach to managing K8S manifests.
Pulumi - Allows you to use “real” programming languages to manipulate cloud environments, including K8S. Interestingly supports multiple programming languages. “Define infrastructure in JavaScript, TypeScript, Python, Go, or any .NET language, including C#, F#, and VB.”
Ballerina - _ “Ballerina is an open source programming language and platform for cloud-era application programmers to easily write software that just works.”_ Write code in the ballerina language that can build K8S objects for you.
Kapitan - “…a tool to manage complex deployments using jsonnet, kadet (alpha) and jinja2. Use Kapitan to manage your Kubernetes manifests, your documentation, your Terraform configuration or even simplify your scripts.” Another tool wrapping jsonnet.
Ansible - Ansible can manage K8S objects and is also fairly good a templating, allowing you to easily write custom inventories and pull in metadata about your systems. Simple and powerful.
Terraform - Terraform can manage K8S objects. Read about why one would use Terraform and K8S here. The fact that Terraform is declarative would likely be a good match for K8S.
Habitat - “Chef Habitat is open source software that creates platform-independent build artifacts and provides built-in deployment and management capabilities.” I can’t do Habitat any justice here, other than to say my impression is that it is a can manage applications on several different types of platforms, of which K8S is one.
Skaffold - “Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters.” Aimed at the inner dev loop, but can deploy to K8S as well.
Kuby - “Kuby wraps kubectl and provides commands like kuby prepare to replace variables in YAML or kuby deploy to prepare and apply deployments to Kubernetes.”
Dekorate - “The are tons of tools out there for scaffolding / generating kubernetes manifests. Sooner or later these manifests will require customization…Using external tools, is often too generic. Using build tool extensions and adding configuration via xml, groovy etc is a step forward, but still not optimal.” I think that introduction is apropos given this list. Keep manifest generation as part of your development/language environment, makes sense to me especially if your org is focussed on one language, i.e. Java.
BONUS: Kpack - A system for using Cloud Native buildpacks. Buildpacks are a more structured, opinionated way to build container images. Joe Beda ran a TKIG on it. The pack command line can be used to build local images, or kpack can be deployed into K8S and will automatically rebuild images when code is changed in repositories. Kpack is focussed in image builds, not so much on generating manifests. (NOTE: Built by Pivotal, now VMware.)
BONUS: Any programming language - I wanted to make sure that I put this on the list, as ultimately any language that can print to standard out could be used to generate K8S manifests. Of course, we would likely not want to do something that simplistic, but I don’t think it’s far fetched or unreasonable to use templates and a generic programming language to build manifests.
As I mentioned at the start this list is random–the order it comes in was the order I discovered them on the Internet. (Certainly there are some cool tools I have missed, let me know in the comments.)
Ultimately, these tools provide some kind of templating, wrapper, abstraction, or other convenience around K8S, though, of course, they all do it in different ways and chose different approaches. The differences and value of these individual tools is typically under the surface layer and thus require some investigation to truly understand when, or when not, to use them.
With so many options it does seem like the market (made up of us technical people–the people who want and use K8S) hasn’t decided how exactly to use Kubernetes. A “silver bullet” approach probably won’t work. Will one of these tools win out? A handful? Will we get something new? Or does Platform as a Service still make the most sense? Time will tell. Until then, we all have to perform some due diligence and keep innovating.
PS. Once I found ~20 I stopped. A few minutes later I found this spreadsheet with 120 similar tools!