13 November 2024 Mannheim, Germany
Nixing Challenges of Kubernetes Packaging with Nix
Abstract
Software supply chain management (SSCM) systems should provide software bill of materials (SBOM) and auditability as well as scanning for vulnerabilities and licensing conflicts. Furthermore, it should offer flexible configuration options and the ability to define comprehensive specifications e.g. for regulatory compliance and every change should undergo a quality assurance (QA) process.
These requirements are addressed by Nix, a functional language and package manager allowing to create reproducible, declarative, and reliable builds. We present a packaging of Kubernetes manifests based on Nix and show how this enrichs the features of SSCM and improve reliability and operational safety.
Transcript
Nixing challenges of Kubernetes packaging with Nix
Hello everyone, and welcome to my talk!
Today I want to show you what is wrong abount Kubernetes packaging and how we can overcome these issues with a technology called Nix.
echo $(whoami)
The slides of this presentation can be found online by scanning this QR code or visiting the respective URL here.
Before we start, let me briefly introduce myself. My name is Arik Grahl and I am a Software Engineer working at SysEleven. SysEleven is a Berlin-based hosting company and we operate a few data centers in Germany and benefit from our own network infrastructure. At the core is our OpenStack-based cloud offering with a managed Kubernetes on top of it.
My team develops the for the end user highest abstraction layer, which is a software supply chain management system with application lifecycle management using our infrastructure. So my everyday business involves Golang development of Kubernetes controllers and of course packaging with Nix.
If you want to reach out, you can contact me via any of these channels.
Kubernetes
Let’s start today’s journey with Kubernetes. Kubernetes is a truly amazing system and I am convinced that it deserves the title of a distributed operating system. However, from a user’s perspective, the system is an API and also components on top of are mostly API-centric. Therefore the execution paradigm evolves around an operator pattern, where resources get reconciled. The overall system state converges eventually towards the by the API defined state.
Under the hood Kubernetes leverages strict Go types and Protobuf, which becomes very evident if you develop components, which are very close to Kubernetes. However, for end-users this API is almost exclusively exposed as a structured document like JSON or more commonly YAML.
YAML
There is nothing to be said against defining YAML as a format for an API. As a format beyond the transport protocol, though, it is not suitable for managing resources.
Especially for complex documents it becomes evident, that it is very verbose and quite repetitive. It is not easily possible to extend the document and organize it into fragments. The provided concepts for abstraction and reusability are not sufficient enough, which result in challenging inheritance. It is certainly debatable whether YAML is a suitable format for structured documents in general. Some type confusions contribute to this reputation.
Tooling to Generate Structured Documents
So far this is no secret and for most users it is evident, that pure YAML doesn’t get them very far. Then something happens, which is often observable in IT: Instead of adressing the root cause of the issue, there is an mitigation of the problem by introducing another level of indirection.
As a result a lot of tools emerge, which deal with the generation of JSON or YAML: Jsonnet, Kustomize, Helm and Helmfile.
Helm (1/2)
In this list Helm deserves special attention, because it evolved somehow to the defacto standard for packaging in the Kubernetes ecosystem. Whenever there is a cloud-native application, chances are incredible high, that you will find a so-called Helm chart, which is their terminology for packages.
It even offers some rudimentary dependency management, by referencing downstream charts. As a lot of applications from the cloud-native ecosystem Helm is written in Go, which comes with its templating implementation as part of its standard library. While I would not challenge this decision on technical level, in the end YAML is plaintext, I would challenge the general approach of templating a structured language.
Helm (2/2)
This leads to templates like this. You often see that those files are introduced with a condition and the templating language dictates a prefix notation, which takes some getting used to, especially for conditions like this.
Definitely worse is the fact, that the template implements some sort of validation here, which violates the separation of concerns: Why should a template do some configuration validation during render time? The plain YAML parts are again ok, within the previously discussed limitations, while it is definitely awkward specify the indentation, whenever a value is injected.
Since this is in the nature of templating a structured document, this is not only necessary for includes, but also for inline YAML content. In particular annoying is, that those if conditions, we talked about earlier, span across hundreds of lines of code, since they guard the entire file. It is way too easy to just miss one of those.
Entering Nix
Luckily, there is a solution to all of this madness.
Let me introduce you a technology called Nix. The claim of Nix is, that it enables system, which are reproducible, declarative and reliable. Let’s find out, what this means and how we can make use of it, to overcome the previously introduced challenges.
What is Nix?
Before we dive into it, let’s differente the terminology here. Nix is a hopefully overloaded term and for newcomers it is usually the first obstacle on their Nix journey is to find out the differences between those pillars.
Nix often refers to the Nix domain specific language or short Nix DSL. This is a purely functional language, where any valid piece of Nix code is an expression returning a value. It also means, that when evaluating a Nix expression it yields a data structure and does not execute a sequence of operations as other imperative languages would do. The language is also lazy evaluated, meaning expressions are only evaluated if its result is actually requested. This makes the partial evaluation of large codebases fairly quick, while on its downside only those errors surface, where the respective execution paths are taken. It is needless to say, that as a domain specific language it is purpose-built. Although used otherwise, it is explicitly not a general-purpose language and is soley designed as a language for the Nix package manager.
This brings us to the second pillar, which is Nixpkgs. It is a package manager and provides respective utility functionality all powered by the language Nix. Nixpkgs is organized in a large mono repository on GitHub, representing one of its largest repositories both in terms of contributions and numbers of contributors. The repository contains all dependencies for a complete and self-contained build chain. You will find an user-space like busybox or coreutils, compilers like GCC and LLVMs as well as shared libraries like glibc or musl. Also Linux kernels and their respective modules are available in the repository. Since this build chain and all components are included and therefore versioned self-contained, it allows reproducible builds, even for arbitrary historic snapshots. Nixpkgs is large, very large: With more than 100k packages it easily outsizes other established repositories. You will find among classic system packages also a variety of language specific packages such as packages for your JavaScript, Python or Ruby project.
The third Nix pillar would be NixOS the functional linux-based operating system with an immutable design and an atomic update model, which builds upon Nixpkgs. Since with Kubernetes we already have an operating system in place, NixOS is beyond the scope of this talk.
Nix as a Generic Builder (1/3)
With the terminology in place we can now have a deeper and more practical look at Nix.
Since Nix is the language for the Nix package manager the goal is usually to build some software and therefore create artifacts.
We will now have a look at a real world example, which is how we would package the well-known commandline program kubectl.
The syntax may look a bit confusing at first, but is very simplistic and it usually doesn’t feel like a programming language. For better reusability, we encapsulate our Nix expression in a function. This function receives an attribute set with the key pkgs, which defaults to the system’s Nixpkgs. Our function returns a derivation, which translates to the artifacts of the subject, which is packaged. Nix provides therefore generic builders, virtually for every language. We are using buildGoModule here, because kubectl is a Go project. The derivation receives an attribute set, defining the input to build kubectl.
Therefore, we would specify the package name and its version. We want to build kubectl with the sources from GitHub, therefore we are using the derivation fetchFromGitHub to receive it. Just like we did with our derivation buildGoModule, we are passing an attribute set to fetchFromGitHub, this time to specify the exact content. This would be the repository’s owner and the name of the repo itself, which is the same as the owner for this case. As a revision we are using the previously defined version, prefixed with a “v”. Nix is very strict for remote content, therefore it requires a hash corresponding to this specified GitHub source code artifact. The purpose is more than just an integrity check and is necessary to ensure reproducibility. Since we have included the complete Kubernetes software project, we would specify the subpackages, which we are interested to build.
The Kubernetes Go project does the vendoring of all its dependencies for us, which is why we should set the vendor hash to null. Other projects, which do not vendor its dependencies as part of the project, would be guarded by a specific hash similar to the fetchFromGitHub hash. This is in fact all we need to reproducible and fully declarative build the kubectl commandline program. If we would build this, we would eventually receive a directory containing the respective binary. Nix conveniently takes care for us, how to build the software making the process as generic as possible. Building other software even in completely other languages would be very similar.
Nix as a Generic Builder (2/3)
While building software and producing a binary is certainly the most common use-case, we are not limited to it at all. We can generate any content, including structured documents, including YAML. The next example should build an Nginx Kubernetes deployment and hopefully highlight, why this is already way superior to Helm and its templating.
Again, to improve reusability, we are encapsulating our package in function and defining pkgs as an argument, defaulting to the system’s Nixpkgs. Local variables are defined with let in Nix and have always a defined scope. Here we defining the name of our package and its respective version. In Kubernetes it is quite common to define the application’s name as part of its labels. We can conveniently use the previously defined variable name for the key app of our attribute set, holding all potential labels. Just like our previous example, our function returns a derivation, which is this time a plain text file indicated by writeText.
With lib.generators.toYAML we can convert an attribute to a respective YAML string in a fully generic fashion. In this attribute set we would then specify our payload in the same structure as we would for any other structured document. For instance a respective apiVersion and kind. We can now use Nix to make the definition of the manifest’s metadata, way less verbose and repetitive.
With the keyword inherit we inject the key value pairs name = "nginx" and labels = { app = name; }, which we have defined earlier as local variables, inside the attribute set of metadata. Even for the selector.matchLabels as part of the manifest’s spec we re-use this labels by simply referencing them. Once more we can re-use the labels by letting the template’s metadata attribute set inheriting it.
The definition of the containers is also pretty straightforward: We define a list containing attribute sets. With inherit we can inject the key value pair name = "nginx" just like we did for the metadata earlier. The image is then a string interpolation assembling an OCI URI containing the previously defined name and version.
It may be a very simplistic example, but I find it pretty evident, that Nix as language brings already a lot to the table by abstracting away repetitive parts in a reasonable extent. Even without specifying the current level of indentation. Building this expression would result in a single file deployment.yaml, with the expected output.
Nix handles Dependencies
We also can compose Nix expressions like this to larger application stacks with dependencies. In this example we are again defining a function like we did in the previous examples. As local variables we are defining the name of our application stack and its version. Also as a local variable we would now define the first dependency, which would be a PostgreSQL database for instance.
By using the import keyword we can include any Nix expression and would evaluate it. The Nix expression in this file could return a derivation like writeText, which we have just seen in-depth in the previous example. The second dependency would be the application itself, which we store under the local variable webapp. I just said, that the import keyword evaluates the respective Nix expression behind it. This means, if this Nix expression in a file is a function, we can also pass an argument to it. Therefore, it behaves like a function call to a function in a different file. In this example we would pass the key value pair version = "1.2.3" to the function defined in the respective file and making it aware of this specific version we are interested in.
Also this Nix expression returns a derivation, just like all of the examples before. This time we are using linkFarm to generate a directory holding symbolic links to downstream artifacts. The first symlink would have the name deployment.yaml and references the output of the webapp we have defined earlier. The output of the database dependency, also exposed as a local variable, would represent the second link with the name statefulset.yaml.
This may again be a very simplistic example, but it shows, that Nix is able to compose other derivations with very little overhead, but still a lot of flexability. Building this expression would result in a directory with symlinks to the contents of the downstream generated YAML files.
Profit from the Established Ecosystem
It may seem a bit like reinventing the wheel, since we are dealing with low-level Kubernetes manifests. Well, the good news is, while we have the flexibility, there is no immediate need to do so. For instance, there is an amazing project called kubenix, which basically offers us utility functionality for all the primitives we know from the Kubernetes core API. This would make our Deployment or Pod example from before even slim, while providing the same flexibility. In fact, it would even add a type system, which would prevent us from specifying values of a wrong type and ensure, that all outcomes are valid Kubernetes manifests.
There is, by the way, no guarantee with Helm for this and we actually have to pass it to the Kubernetes API server, to find out if the manifest is valid. Another interesting feature of kubenix is its Helm wrapper. I spend a solid ten minutes stressing that Helm is a terrible tool to package for Kubernetes, but it is also the de-facto standard for packaging cloud-native workloads. When consuming a lot of downstream cloud-native projects, it would not be wise to ignore this fact completely. However, this Helm wrapper yields Nix expressions once again, so we get the same flexibility and benefitting from Helm as an ecosystem.
Even if you rely on some other special tooling, Nix may the means of choice. Inside your derivation, you can specify any building script you like, which is basically just a shell script. Let me highlight this real quick with a very small example. We specify a function returning a bare derivation and using the currentSystem as our system here. As the name we are using Kube Prometheus Stack with a certain version, which we want to package here and where we rely on a custom tool like helmfile for whatever reason. We are using fetchTarball to get the required sources, which is very similar to the fetchFromGitHub, we have seen earlier. As a builder, we can now declare a shell script with writeShellScript, which will be executed to generate our derivation’s output.
Inside the shell script, we can access any variables and reference dependencies like helmfile itself in order to produce our desired output. This will be written to the pre-defined variable $out. To sum it up, we can profit from established ecosystems by using kubenix, in particular its Helm wrapper and do not need to sacrify any flexibility with custom builder scripts.
Containers (1/2)
So the problem of structured documents seem to be solved as we seem to have tamed YAML and outperformed existing tooling with Nix. However, beyond that, we haven’t gained much by generating YAML. In particular, we have the very same problems concerning our software supply chain security and management.
Also a structured document representing a Kubernetes manifest does not represent a software bill of materials. When looking at a Kubernetes manifest of a Pod, for example, we see, that the interesting bits are hidden behind an OCI URI, which is just a reference to an artifact sitting somewhere. Wouldn’t it be cool if we somehow could overcome this limitation?
Containers (2/2)
As a matter of fact, Nix is able to do exactly this. Bear with me in this last Nix example illustrating this use-case. One final time, we are defining a function and specify the name of the package we are defining. This is intended to be a Kubernetes manifest of an Nginx workload, with a self-contained container.
As its version we are using the version of Nginx, which we find in the respective Nixpkgs. Remember earlier? Nixpkgs a software repository consisting of over 100k packages. Our next intermediate goal would be to get Nginx from Nixpkgs into a container. Conveniently, there is a derivation dockerTools.buildLayeredImage available, which depsite its name builds OCI containers completely without Docker. This is great, because containers when built with Docker are far from being deterministic let alone reproducible. With inherit we give this container the respective name and as a tag we are using the previously determined version, suffixed with an “v”.
Instead of writing a Dockerfile, which would build a container in an imperative fashion, we simply declare its content. Therefore, we specify as an Entrypoint the Nginx binary from the corresponding Nixpkgs package. Nix will take care of including it and its depedencies in the container for us. In addition, we specify common command line arguments, which with the container should be started per default. And just like that we have built an OCI container without Docker in a fully declarative and reproducible fashion.
Also, every dependency, which for instance a shared library like glibc translates to a container image layer. Unlike other container builds, this gets the best out of container image layer caching, just as it should be. And did I mention, that those containers are in total significant smaller than most other containers? When building this, we would get a gzipped OCI image, which we could load with a container runtime of our choice.
So far so good, we now have the container, but originally we wanted to create a Kubernetes manifest. Just like in the examples before we return a derivation and generate a text file with writeText. For sake of simplicity, we will generate a plain Pod. Again, we are inheriting the name and specify our containers, where again the name is inherited. Now, for the image everything comes together: With string interpolation we are crafting an OCI URI with the custom domain nix.internal, but we can theoretically use everything. The previously defined container is also referenced in this string interpolation, resulting in a unique content-based hash. We developed a very simplistic, but fully compliant read-only OCI registry, which acts like a proxy and serves the implicitly built OCI container images, which are just gzipped tar balls in the Nix store.
By this, we can finally define containers as strict dependencies for its Kubernetes manifests. This is hugh gain: The whole industry, on the other hand, relies of some potentially completely independent process of generating those artifacts and hope that those artifacts match its references.
End-To-End Experience
This leads to a very relaxed and coherent workflow, which I often describe as an end-to-end experience. We build with Nix, completely declarative and reproducible. As a result we get a directory containing Kubernetes manifests, which guarantee of having strict dependencies to its container images.
There is nothing more to do, than simply applying this directory with kubectl. That’s literally it. We are using exactly one tool to build everything: There is no Golang, no Docker, no Git, only Nix. We get reproducible and pure builds absolutely free, thanks to Nix. It will work on every machine and every machine will produce the same output.
It also enables a very GitOps friendly workflow, if this is your cup of tea. Just check in your Nix expressions, representing your Kubernetes manifests and execute those two commands as part of your CD pipeline. Since it is so simple to use, but yet so powerful, it is very flexible when used together with custom tooling.
Detailed Insights (1/2)
This end-to-end experience is very useful, but the original motivation was to gain detailed insights beyond the structured documents of Kubernetes manifests. Since Nix is already very strict about its dependencies, we have this information already.
For the previous example, we can for instance simply run nix derivation show --recursive ./result, do some selection and aggregation and we will get all of its dependencies, wether it is for building or running the software.
Detailed Insights (2/2)
We can visualize this information more appealing and create a dependency graph, like I did for wget here. In fact there is a rich tooling in Nix and I want to briefly highlight sbomnix here, which feels like “one stop shop”. It can generate SBOMs fully automatically and also scans for known security vulnerabilities in it.
One can query for licenses in order to find potential incompatibilities for a specific business case and even provides tooling for provenance attestation. Other related tools would be nix2sbom to generate SBOMs or vulnix to scan for security vulnerabilities.
MetaKube Accelerator (1/2)
While we are big fans of Nix at SysEleven, for our common end users we have a different vision. We want to abstract away most of the Nix tooling. Our product MetaKube Accelerator therefore is a cloud-native distributed application consisting of a varienty of controllers running on Kubernetes.
Therefore, we invision an API-centric approach enabling the kubectl-first experience for end-users.
MetaKube Accelerator (2/2)
To conclude, I want to present the architecture of MetaKube Accelerator on a very high level. We distinguish between a provider cluster on the one hand and a tenant cluster on the other. The provider cluster is operated by SysEleven and exposed via a publically reachable endpoint. The tenant cluster can be any certified Kubernetes distribution and does not need to be publically exposed. At the core of the provider cluster is a Nix service, holding Nix derivations, which were built very similar to what I presented. In this example we have the derivations for ingress-nginx, cert-manager and external-dns.
Throughout a continous process we synchronize those derivations to Kubernetes custom resources, representing our building blocks. In another continous process on the tenant cluster these custom resources will get synchronized eventually on the tenant side. This enables an end user looking at the tenant cluster to browse the software catalogue of building blocks, just with kubectl get. Through a different custom resource the end user marks one or more of these building blocks for installation by creating said resource with kubectl create. This installed state is represented in turn on the provider side in a Nix profile.
The drift reconciler is an agent running on the tenant side, continously observing the user-specific Nix profile. As the drift reconciler’s name suggest, it will ensure, that the cluster state is in sync with the corresponding Nix profile and therefore apply or delete respective manifests. When the tenant wants to modify a building block, for instance providing a different configuration for ingress-nginx, it specifies so by means of another custom resource. The tenant creates this custom resource with kubectl create and specifies the already existing building block, on which it wants to build on, along with the desired modification. During reconcilation of this custom resource, on the provider side, a new derivation is built in the Nix store. It will be converted again to a corresponding custom resource, but this time it indicates, that this building block is owned by the specific tenant. It will get synchronized in the same way as before and can therefore be marked for installation throughout this respective custom resource.
Hereby the Nix profile on the provider side is mutated, reflected the new state and the changes will be finally picked up by the drift reconciler and rolled out in the tenant cluster.
Conclusion
Finally, let’s sum up today’s insights. We have seen, that defining Kubernetes manifests with Nix provides a resonable amount of abstraction thanks to the domain specific language. It even enables sophisticated dependency management.
While we are abstracting the problem space, we can still build on top of the established ecosystem.
Questions & Answers
This brings me to the end of my talk and I hope you liked it.
Once again, you can find the link to this talk’s slide behind the first QR code and you can learn more about our product by scanning the second QR code. I am curious what you say and I am happy to answer some questions.