OpenStack Europe October 2024

17 October 2024 online

NixOS: A Brief Introduction

Abstract

This talk introduces NixOS and its fundamental concepts, clarifying the distinction between the Nix language (a functional, lazy, dynamically typed DSL for package management), the large and up-to-date Nixpkgs repository, and NixOS, an immutable Linux operating system with declarative configuration, atomic updates, and easy rollback capabilities.

Through practical examples and demonstrations, the unique workflow of NixOS is highlighted—including its management of system generations and reproducible builds via the Nix store and symlinks. The creation of deterministic, minimal container images with Nix is also demonstrated, emphasizing advantages over traditional Docker builds.

Attendees receive a concise overview of how Nix, Nixpkgs, and NixOS streamline package management, system configuration, and containerization.

Transcript

Intro

Hello everybody and welcome to my talk!

Today I want to give you a brief introduction on NixOS and its basic principles.

echo $(whoami)

This presentation can also be found online by scanning this QR code.

Before we start, let me introduce myself: My name is Arik Grahl and I work at SysEleven. SysEleven is a Berlin-based hosting company with datacenters in Germany and a dedicated network infrastructure. We have an OpenStack-based cloud offering and a managed Kubernetes offering on top of it.

My team develops the highest abstraction layer, which is a software supply chain management system and application lifecycle management on top of this stack. My daily business mostly deals with Kubernetes controllers and operators which we develop in Golang and of course packaging with Nix.

If you like to reach out, you can contact me via any of this channels.

What is Nix?

Nix is a hopelessly overloaded term. Therefore, let’s start by answering the question what Nix actually is. In this figure you can see, that Nix is often used interchangably for the language itself, which I will call “Nix DSL” to make it more clear. On the other hand the term Nix often refers to Nixpkgs a large package ecosystem on which NixOS is built. Even for this operating system NixOS itself the term Nix is occasionally used, which is confusing in particular for newcomers.

We will now turn our attention to these three pillars of the Nix trinity.

Nix DSL (1/3)

The Nix language is a purely functional language. This means, that any valid piece of code is an expression returning a value. When evaluating a Nix expression this returned value will always yield a data structure. In contrast to imperative programming languages, there is no sequence of operations to be executed.

Let’s have a look at some examples, to get a rough idea about this language. By the way: You can try out almost all of the examples in this presentation in a nix repl.

When we are evaluating the string "foo" the same value of it will be returned. It is the same for more complex data structures: Now we are evaluating the attribute set { x = "foo"; }, which will return the very same data structure. We can define a variable f here, with the static value of the string "foo". When evaluating this variable it will return the string “foo”. Now, let’s store an anonymous function under the variable f. It receives the argument x and returns the very same argument. When calling the function behind the variable f with the string "foo" as an argument, we will receive its value back. You might have already noticed, that there is no notion of strict types in this examples. This is why Nix is dynamically typed, meaning, we might as well pass the attribute set { x = "foo"; } into the same function exposed via the variable f.

Another interesting property of Nix is, that every function is unary, which means it accepts exactly one argument. In order to construct functions, which consume two or more arguments, we make use of a concept, which is called Currying: We use a higher-order function, which receives another unary function as an argument. In our example we are implementing a concatenation with a space of the arguments a and b. For the caller the functions behind the variable f look like a single function with two arguments, where the strings "foo" and "bar" are passed. While this is common in Nix and even more idiomatic way of passing a series of arguments, is by encapsulating those in an attribute set. One last time we are using the variable f to store an anonymous function, which receives an attribute set with the keys a and b. Otherwise it is the same implementation of our concetanation with space from before. We can now call this function with a respective attribute set as an argument, where we are assigning the strings "foo" and "bar" to the keys a and b, respectively.

Ok, I promise, that was already the most complicated slide regarding functional programming and it won’t get any worse.

Nix DSL (2/3)

The second property of the Nix language is, that it is lazy evaluated. Meaning that only this paths of the expressions are actually evaluated where its results are requested. This is crucial for large codebases like Nixpkgs, where you usually only want to build a certain package and not the entirety of packages known to humanity. However, the disadvantage lies in the fact, that some errors only surface, when the respective execution paths are taken.

This example illustrated lazy evaluation: We have an attribute set with the name a, which stores under the key x the static string "foo" and throws an exception for the evaluation of the key y. When evaluating the key x in the attribute set, we will successfully receive the string "foo". On the other hand, when accessing the key y of the very same attribute set, it will crash with the respective exception.

Nix DSL (3/3)

The language Nix is purpose-built. It means that it is a domain specific language, serving the Nix package manager. It can be used beyond that, but it was never designed to be a general-purpose language.

Don’t be frightened, I will now show you a real-world application of the Nix DSL. You are not intended to understand it in-depth, it should just get you a feeling how the language is actually used. This is the packaging of setup-envtest a common project used for testing Kubernetes interactions.

Nixpkgs (1/3)

Now since we have an understanding of the underlying language, we can take a look at the primary purpose it is serving, which is the Nix package manager. Nixpkgs is often used synonymously with the package manager, which is primarly a large mono repository on Github. In fact, it is one of the largest repositories in terms of size, contributions and contributors.

The mono repository approach makes it easy to define a complete build chain for building all different kind of packages. This is a basic user space like busybox or coreutils, a compiler and also dependencies in form of shared libraries up to sophisticated kernel modules. Since all components are included and therefore versioned self-contained, it allows reproducible builds, even for arbitrary historic snapshots.

Nixpkgs (2/3)

Nixpkgs is large, very large: With more than 100k packages it easily outsizes other established repositories. You will find among classic system packages also a variety of language specific packages such as packages for your JavaScript, Python or Ruby project.

Here you can see the top 10 package sets included in Nixpkgs, reaching from scientific purposes (R, Haskell, Python, OCaml) to editor specific packages (Emacs, Vim). At least in terms of amount of packages Emacs seems to dominate over Vim.

Nixpkgs (3/3)

Given the gigantic size of the repository one might assume, that the majority of the packages is rotting away. Surprisingly, this is not the case and most of the packages are pretty up-to-date.

For well-known repositories, this visualization shows the number of packages compared to the number of fresh packages. In the top right corner, you will find nixpkgs unstable and next to its versioned variant nixpkgs 24.05, which is slightly less up-to-date. The Arch User Repository is the closest in terms of amount of packages but has significantly fewer fresh packages than Nixpkgs. The cluster Debian Unstable and Ubuntu 24.10 is fairly fresh, but has roughly only a third of the packages as Nixpkgs. Another cluster with even lower amount of packages, but reasonalbe freshness, would be Fedora 40 and Gentoo.

NixOS (1/4)

With the powerful package manager and ecosystem in place, we can now have a look at the operating system, which builds on top of all this: NixOS. It is a linux-based operating system with an immutable design and an atomic update model. With such ambitious and modern design principles, I find it quite remarkable, that it was already released over 20 years ago.

It supports major CPU architectures such as the 64 bit versions of x86 and ARM. There is experimental support for other architectures, but since it lacks a binary cache for built artifacts, it is not too much fun to use. It is licensed very permissively under MIT, which permits usage within proprietary software.

NixOS (2/4)

NixOS has many benefits, but in my opinion the unique selling proposition revolves around the single configuration file that describes the entire system. When I say single configuration file, I already lied to you, because usually a NixOS system is composed of two configuration files. For practical reasons, one is a hardware-specific file, which is most commonly generated during installation and will typically not be modified, unless the hardware of the system changes. The second configuration file is hardware agnostic and mostly deals with generic service definitions.

Let’s get quickly over a more or less minimal example of such a configuration file, to get a rough idea how such a file can look like. Through the Nix module system the hardware configuration file is imported. We define an EFI system with the systemd bootloader. As we will see later, treating the booloader as a first class citizen in the operating system brings a lot of benefits to the table. We are using the latest standard kernel, which comes with NixOS. Batteries are included, it will come with the needed kernel modules defined by higher level service definitions, such as hypervisor definitions or alike.

Setting up the timezone system-wide is done – as everything in NixOS – fully declarative and other localization options would look similar to this. Usually, in NixOS enduser software is not installed directly, as you would normally do in other Linux operating systems. Instead, you define the desired set of enduser software packages to be installed fully declarative. In this example only the editor vim is installed.

The installation of software in NixOS usually follows a completely declarative approach by defining higher level services. Instead of defining software packages to be installed, we define the capability or service behind it. If we want to be able to run containers on our system, we simply set virutalization.podman.enable to true. Necessary kernel modules will be loaded and the necessary userspace programs will be provided automatically. Last but not least, we specify the stateVersion, which reflects the entirety of the options available in our NixOS configuration and is independent of the actual software versions.

NixOS (3/4)

NixOS options are very powerful and provide the declaration of a cascade of complex services. Very useful is the property, that the lifecycle of the defined services is managed automatically by Nix. It is almost never necessary to manually invoke a systemctl restart after a new configuration is rolled out.

This services section outlines a definition of a system, which is exposed via SSH and runs a Kubernetes node. It is quite idiomatic to specify enabled = true in order to activate the service. Other configuration like to permit the root user to login via SSH or the role of the k3s node can then be specified in the respective service definition as well.

NixOS (4/4)

In NixOS an update, a installation of a new package or a reconfiguration of a software all lead to a generation. Since everything is declarative with nixos-rebuild one can re-apply the current configuration. If there were no changes to it, the system stays unaltered. To receive updates one would either update the respective repository channel or use the `–upgrade´ flag to do it automatically.

Another interesting attribute is to decouple building and rolling out changes. With the verb switch you would directly roll the changes out, once they are built, thus making them available directly. The verb boot on the other hand, just builds the changes and marks it a boot target for the next reboot. Remember the integration of the bootloader as a first class citizien, which I mentioned earlier? NixOS exposes every generation as a selectable boot target. So every change can be easily and safely reverted but simply booting into the previous system state. Generations can also be explicitly rolled back with the corresponding flag in nixos-rebuild. Since those generations and lots of old versions tend to pile up over time, one can garbage collect all unreferenced artifacts or delete them based on a time based policy.

Demo: NixOS (1/5)

So let’s have a more practical view at NixOS. In order to save precious time waiting for commands to finish, I will present my demo here in this presentation. The first use-case we are looking at, is to update our system, in particular we want to perform a kernel update. Intially we are on version 6.11.0.

Since a kernel update can’t be performed online with NixOS, we want to boot into the kernel after the next reboot. Therefore, we are using the verb boot when invoking nixos-rebuild. To check for new versions online and not simply rebuild the very same configuration we are having before, we are using the flag --update. As specified it is pulling the channels for updates and building the new system configuration.

With nixos-rebuild list-generations we can see, that there exists a new generation, with a new kernel, which isn’t active yet.

Demo: NixOS (2/5)

We then reboot our system and will be prompted with a similar overview of generations in the bootloader. After the system is up and running again, we re-run nixos-rebuild list-generations and can see, that now the new generation is marked as current.

Re-checking the current kernel verifies, that this update was successful.

Demo: NixOS (3/5)

The second use-case we are looking at is to install new software. On a higher level we want the system to become a Kubernetes node. First, we check if it already runs Kubernetes, by executing kubectl get nodes and see that not even kubectl is installed.

Instead of installing all the Kubernetes related packages, configuring and bootstrapping the cluster, we go back to our initial requirement: The system should become a Kubernetes node. This is what we are adding to our Nix configuration. For sake of simplicity I am using the lightweight k3s distriubtion, but a vanilla Kubernetes is also available as a Nix option.

Demo: NixOS (4/5)

Now it is time to apply this configuration. We are impatient and want to avoid another reboot and directly start the Kubernetes node, that’s why we are using the verb switch when running nixos-rebuild. Since we have just updated our system, there is also no need to specify the --upgrade flag again. You might have noticed that in addition to the building of the system configuration NixOS set up the /etc directory and started the systemd unit k3s.service.

Now we are running nixos-rebuild list-generations again and see, that we are directly in a follow-up generation. Since we haven’t updated the NixOS version for instance stays the same.

Demo: NixOS (5/5)

To verify our configuration, we re-run the command kubectl get nodes and indeed get a successful response. NixOS hasn’t just conveniently provided the kubectl CLI, but more importantly has configured and started a Kubernetes cluster.

A production-ready multi-node cluster wouldn’t be even significantly more complex.

How does Nix work? (1/4)

You might ask yourself: How can Nix possibly deal with such complexity? The answer lies in the fundamental theorem of software engineering, which states:

All problems in computer science can be solved by another level of indirection.

This theorem is postulated by the british computer scientiest David John Wheeler.

How does Nix work? (2/4)

Let’s illustrate this with a simple example: The binary of the well-known application rsync. We can see that Nix exposes it in a quite unusual path.

More interestingly, if we follow this symlink, we see that it is acutally pointing somewhere else, in the Nix store. This is the central structure, where all artifacts known to Nix are stored.

All artifacts? What about configurations like the weird path? As we can see, this directory is a symlink to a Nix store artifact of a system path.

And what is /run/current-system then? Again a symlink to a Nix store directory representing the generations we have seen earlier.

So, to sum it up: Our simple rsync binary is a cascade of three symlinks.

How does Nix work? (3/4)

The next question is: Why stop with the indirection at the binary? Can we go deeper? Sure, we can. What about the dynamically linked shared libraries like glibc and alike? If we list those for our binary, we see that all this dynamically linked shared libraries are again references to artifacts in the Nix store.

This makes NixOS both very flexible in using all different kind of library version combinations as efficient in reusing shared libraries among different binaries.

How does Nix work? (4/4)

This brings us to the second – less famous part – of the theorem, which adds:

Except for the problem of too many layers of indirection.

In fact, Nix is doing a great job to handle this and fully abstracts it away for the user.

Container

I want to continue my talk with another interesting use-case of Nix beyond the previously highlighted ones, which is to use as a container builder. You might wonder why not to stick to the established traditional Docker build.

Generally, for most users there is nothing wrong about using Docker to build containers. However, you should be aware, that it is not even close to being deterministic. Most people don’t notice it and in the majority of the cases it doesn’t matter, but in some situations it might be crucial. Another downside is, that often a package manager and some Linux distribution repository is used to compose packages inside the container. This is simply a waste of resources and unnecessarily increases the attack surface. You might want to use multi-stage builds to avoid most of this problems, but then the layering and therefore caching of different dependencies is still pretty poor.

Demo: Container (1/3)

Nix can address all of this issues. Let’s find out how.

We are assuming, that you are a Golang developer and have this simple web application. You are not supposed to understand this in-depth, but just let me briefly outline this. It listens on the TCP port 8080 and serves an HTTP server there. For every request it calls a downstream server via HTTPS to retrieve the weather data for Berlin.

Demo: Container (2/3)

Nix brings convenient functionality to build software for all kinds of languages, which also applies for Golang. We specify a package name and a version. As the source input we are using the current working directory, where our Go code lies. Since we are good with Golang’s standard library and do not need dependencies, we can specify an empty vendorHash. This is all we need to build this binary only with Nix, fully declarative and reproducible.

Wrapping this into a container can be achieved conveniently with the buildLayeredImage functionality. It accepts a name for our container and a tag, which we are setting to the version. You might have noticed, that we are depending on a CA chain to initiate a TLS connection to the downstream service. Therefore we define the package cacert to be part of the container’s content. By referencing the output of our previously defined Golang binary as the entrypoint for the container it gets copied into the container implictily and in a fully declarative fashion.

Demo: Container (3/3)

We then build this Nix expression with nix build and getting a result symlink in our working directory pointing to the final artifact in the Nix store.

This is the container image as a gzipped tar ball and can be loaded with the container runtime of your choice. We can then run this container while exposing the TCP port 8080.

With an HTTP client we can check this endpoint and indeed it is returning the weather report for Berlin.

Conclusion

Let’s sum up today’s contents. Nix is a powerful language, which satisfies different use-cases around packaging and configuration management. Nixpkgs is a large and attractive ecosystem both providing packages and convenience functionality to build such. NixOS is production-ready and flexible immutable operating system with an atomic update model. Building containers with Nix offers both the desired properties of Nix, which not breaking existing tools, thus making it a perfect candidate to start with.