What is stage0?

It is a chain of compilers and assemblers that can be built from source, starting from a 357-byte program that can eventually build a recent GCC.¹

Since then, NixOS and other distributions have also adopted the same approach to minimize their binary seed which makes it possible to onboard new architectures and platforms much simpler.

What’s always frustrated me as a Bazel (& Buck) user is the reliance on prebuilt toolchains even for things that should be built from source easily like protoc.

Bazel has given up trying to provide a hermetic C++ toolchain and the upstream rules_cc ruleset just points you elsewhere:

Configuring a hermetic toolchain makes your build more deterministic. rules_cc itself does not yet offer a hermetic toolchain distribution

I had attempted to provide a stage0 hermetic C++ toolchain in October 2024 via https://github.com/fzakaria/stage0-bazel. I made substantial process through the bootstrap process but I did not make it far enought to be usabale.

To be honest, I was also a little disheartened that no one else in the community thought it was the greatest thing since slice bread. Everyone seems to be content with using prebuilt toolchains as they go deeper into MODULE.bzl madness.

I had put it aside for a while, but I have been thinking about it again recently. The steps are mechanical and the process imitates existing distributions, so this became a perfect project for me to throw at an LLM to finish.²

You can now leverage the toolchain to build cc_binary in Bazel and have it compiled by a toolchain whose entire ancestry is in the repository from that same 357-byte seed. 🎆

How complete is this toolchain?

I pointed the toolchain at Abseil and GoogleTest straight from the Bazel Central Registry without any patches.

bazel_dep(name = "stage0-bazel", version = "0.1.0")
bazel_dep(name = "abseil-cpp", version = "20260107.1")
bazel_dep(name = "googletest", version = "1.17.0.bcr.2")


register_toolchains(
    "@stage0-bazel//toolchain:clang",
    "@stage0-bazel//toolchain:cc",
)

We then can build and run their testsuite to provide a sanity check that the toolchain is working correctly.

$ bazel test --target_pattern_file=absl-tests.txt
Executed 236 out of 236 tests: 236 tests pass

We use a --target_pattern_file to filter tests that require google_benchmark. Abseil marks google_benchmark as a dev_dependency, and Bzlmod drops dev dependencies of non-root modules.

That is us building Abseil and GoogleTest, from the registry, unpatched, compiled by a toolchain that began as 357 bytes of hex.

digraph bootstrap {
  rankdir=TB;
  fontname="Helvetica";
  node [fontname="Helvetica", fontsize=11, shape=rect,
        style="filled,rounded", color="#4C78A8", fillcolor="#DCE6F1"];
  edge [fontname="Helvetica", fontsize=9, color="#666666", arrowsize=0.7];
  nodesep=0.3; ranksep=0.4; pad=0.3;

  seed [label=<<b>hex0</b> &nbsp;<font point-size="9">357 bytes</font>>,
        fillcolor="#F6D5D5", color="#D62728", penwidth=2];

  stage0 [label=<hex1 → hex2 → M0 → cc_x86<br/><font point-size="9">M2-Planet, kaem, M1</font>>];
  mes    [label=<<b>GNU Mes</b> &nbsp;<font point-size="9">mescc</font>>];
  tccmes [label="tcc-mes"];
  tcc    [label=<<b>tinycc</b> &nbsp;<font point-size="9">self-hosted</font>>];
  musl   [label="musl"];
  bin    [label="binutils"];
  gcc46  [label="GCC 4.6.4"];
  gcc10  [label=<<b>GCC 10.4.0</b> &nbsp;<font point-size="9">C++17</font>>, penwidth=2];

  extras [label=<tar, findutils<br/><font point-size="9">Linux UAPI headers</font>>,
          fillcolor="#EFEFEF", color="#999999"];

  llvm [label=<<b>clang 22.1.8</b> &nbsp;+&nbsp; <b>lld 22.1.8</b>>,
        fillcolor="#CFE3D4", color="#2CA02C", penwidth=2];

  absl [label=<<b>Abseil + GoogleTest</b><br/><font point-size="9">236 tests pass</font>>,
        shape=note, fillcolor="#FFF6D5", color="#BF9000"];

  seed -> stage0 -> mes -> tccmes -> tcc -> musl -> bin -> gcc46 -> gcc10;
  gcc10 -> extras [style=dashed];
  extras -> llvm [style=dashed];
  gcc10 -> llvm;
  llvm -> absl [color="#BF9000"];

  { rank=same; gcc10; extras; }
}

How can I be so sure this is a hermetic toolchain?

The toolchain includes an audit report that uses Bazel’s aspects to inspect every action in the build graph and verify that it only executes programs built by the toolchain itself. The report is generated by running bazel build //:trust-report and will fail if any action executes a program outside of the Bazel output tree.³

The report is two lines long:

Bootstrap trust report

Every action in the checked graph runs a program built by this
repository, except for these audited seed binaries:

external/+_repo_rules+hex0-seeds/POSIX/x86/hex0-seed
/nix/store/…-bash-interactive-5.3p3/bin/bash

Unfortunately, since genrule runs a shell it takes as an absolute system path that is also listed as a seed binary. sh_toolchain’s path attribute is a string, and the shell is not a declared input of the action, so no artifact this repository built can provide it.

Building toolchains from bootstrap seeds was never a priority for companies like Google where they control the entire build environment. However we seemed to have adopted the same approach as Bazel and similar build systems have become more popular in the open-source community. We should strive to make our builds more reproducible and hermetic, and this is a step in that direction.

For this to work, the derivation must be a complete description of the build. Anything missing from the derivation causes the reproducibility to break and Nix no longer to be “reproducible”.

In a previous post I built a source-bootstrapped OpenJDK and we had to provide some additional flags:

$ nix build .#openjdk \
    --option filter-syscalls false \
    --option sandbox-paths '' \
    ...

--option sandbox-paths mounts additional paths into the sandbox.

You can see the default sandbox paths on your machine with:

$ nix config show sandbox-paths | tr ' ' '\n'
/bin/sh=/nix/store/zrynrzpsy2993w555ns9a734lbzfff2b-busybox-1.37.0/bin/busybox
/nix/store/cdd109fhy1axl7xb5wisv3v5pd6fawdj-qemu-aarch64-binfmt-P
/run/binfmt

Okay so what’s the point?

Turns out that these sandbox-paths are a hidden input to the derivation. The derivation does not mention it, but it can change the output meaningfully in its repeatability very subtley. 😬

Let’s explore this with a small example.

Here is a derivation with no dependencies. It looks for a file /truth. If it finds one, it believes it. If not, it falls back to arithmetic.

# answer.nix
derivation {
  name = "answer";
  system = "x86_64-linux";
  builder = "/bin/sh";
  args = [ "-c" ''
      if [ -f /truth ]; then read -r x < /truth; else x=4; fi
      echo "2 + 2 = $x" > $out
    '' ];
}

/truth is not in the store, and the Nix sandbox does not mount it, so an honest build never sees it.

$ nix-instantiate ./answer.nix
/nix/store/xbik44ifqm4jqjp4z7n1031smj08mil7-answer.drv

$ nix-store --realise /nix/store/xbik44…-answer.drv
/nix/store/qba6hdgvdrry4k1z83v2zm5xy714l83m-answer

$ cat /nix/store/qba6…-answer
2 + 2 = 4

Our output hash is qba6hdgvdrry4k1z83v2zm5xy714l83m.

We now can add an additional sandbox-path which will cause the repeatibility of the builder to break. We will mount a file /truth into the sandbox that contains a lie.

$ echo 5 > /tmp/truth

$ nix-store --delete /nix/store/qba6…-answer      # throw away the honest one

$ nix-store --realise /nix/store/xbik44…-answer.drv \
    --option extra-sandbox-paths "/truth=/tmp/truth"
/nix/store/qba6hdgvdrry4k1z83v2zm5xy714l83m-answer

$ cat /nix/store/qba6…-answer
2 + 2 = 5

Our output hash is still qba6hdgvdrry4k1z83v2zm5xy714l83m. 😭

The derivation (.drv) is meant to be the complete recipe. Since the sandbox is configured outside the recipe it now causes the same derivation to be leaky and no longer hermetic. Sandboxing is often a way to enforce hermticity and here it actively breaks it.

The sandbox-paths are nowhere to found in the derivation.

This is in contrast with __noChroot, which is a derivation attribute.

$ nix derivation show /nix/store/xbik44…-answer.drv
{
  "…-answer.drv": {
    "builder": "/bin/sh",
    "args": ["-c", "if [ -f /truth ]; …"],
    "env": { "out": "…qba6…-answer", "name": "answer", … },
    "inputs": { "drvs": {}, "srcs": [] },
    …
  }
}

Two people can evaluate a byte-identical .drv, run different actual build steps, and Nix create the same output hash.

You may be thinking “Well, a derivation could have always relied on /dev/random or date, so what’s new?”. True, this is a form of non-determinism. The difference is those are more evident in the derivations to discover and audit for. The act of checking for the existence of a file is extremely subtle and not something that is easy to discover.

In fact, your derivation may appear to even be byte-reproducible on your machine with the --check flag, but it may not be on someone else’s machine.

How problematic is this?

The default value of sandbox-paths is not a constant baked into the source. It is a compile-time property of your particular Nix binary. Two people running nix --version that prints the same number can have different sandboxes before either of them touches a flag.

The /bin/sh entry is not in my nix.conf. It is the default, and it comes from here in the Nix source src/libstore/globals.cc:

#if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL)
    sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
#endif

There are at least two different baselines you can build Nix with, none of them visible to any derivation:

• Built with sandbox-shell set to a path and binary with hopefully the same semantics.
• Built without sandbox-shell such that the option is empty. No /bin/sh at all.

Nix’s own documentation describes this possibility.

Depending on how Nix was built, the default value for this option may be empty or provide /bin/sh as a bind-mount of bash.

Although that itself is not true since Nixpkgs builds Nix with busybox not bash.

(lib.mesonOption "sandbox-shell" "${busybox-sandbox-shell}/bin/busybox")

Given that Flakes have made Nix become incredibly more decentralized, causing us to rely on multiple binary caches, there is no way now to guarantee that the builds are repeatable despite the fact that you can download the binary from the cache.

Is this a bug? 🤔

This is tough to say. Including the sandbox-paths in the derivation would make Nix completely unusable. Suppose you did fold sandbox-paths into the output hash by having it in the derivation. My busybox is busybox-1.37.0 at one store path; yours is a different version at a different path. The same derivation would then hash to different outputs on our two machines, and binary-cache sharing across would collapse.

The property that lets us share builds is the same property that lets us poison them.

None of this is hypothetical and this is how it comes back to the OpenJDK from the top. I hit this hidden input building the OpenJDK via GuixPkgs, which translates Guix’s derivations into Nix and builds them with the Nix daemon.

Guix was forked from Nix much earlier than the commit that introduced --option sandbox-paths and as a result guix-daemon’s build container has no /bin at all, so there is no /bin/sh.

Guix packages are built on the assumption that /bin/sh does not exist. As a result when building the OpenJDK, one of the build scripts failed to have it’s shebang patched and it was expected to fail. Guix tolerates the failure and carries on. Nix, on the other hand, has /bin/sh and the script runs instead of failing, and the build quietly takes a different path which turns out to cause a broken output.

I opened an issue on Guix about this accidental behavior.

Making matters worse, I had already uploaded this broken output to my binary cache. Even when I discovered the sandbox-paths was the issue, I kept generating the same .drv and thus the same output hash, and was substituting the broken output.

Since sandbox-paths is a trusted user setting, I could have maliciously poisoned my own binary cache anyways by signing a broken input. This is only worse in that I did it unknowingly.

You can’t trust code that you did not totally create yourself. — Ken Thompson, Reflections on Trusting Trust

“Specifically I would like to see openjdk translated. openjdk is not bootstrapped from source code in nixpkgs” [issue#3]

Ever since I learned about the stage0 bootstrap chain and how Guix announced full-source bootstrap for all packages in 2023, I was in awe. They provided a package graph of more than 22,000 nodes rooted in a 357-byte program, including the JDK.¹

GuixPkgs can now build OpenJDK 25, which means Nix now has access to a source-bootstrap build of OpenJDK 🎉

$ nix build .#openjdk \
    --option filter-syscalls false \
    --option sandbox-paths '' \
    --print-out-paths
/nix/store/kihk714ljmvhqh36cigc7mfimv0q0rbr-openjdk-25-guix-wrapped

$ /nix/store/kihk71…-openjdk-25-guix-wrapped/bin/java --version
openjdk 25 2025-09-16
OpenJDK Runtime Environment (build 25+-adhoc.nixbld.source)
OpenJDK 64-Bit Server VM (build 25+-adhoc.nixbld.source, mixed mode, sharing)

Does it really work? Let’s take it for a spin.

javac lives in the jdk output, since Guix splits the package.

$ nix build .#openjdk.unwrapped^jdk
$ cat Hello.java
void main() { IO.println("Hello from a source-bootstrapped JDK"); }

$ ./result-jdk/bin/java Hello.java
Hello from a source-bootstrapped JDK

Why a JDK is a hard package

javac is written in Java. HotSpot is C++, but the class library it needs is Java, and the compiler that compiles the class library is Java, and it runs on a JVM that needs a class library… and so on. This is a bootstrapping problem.

Nearly every distribution resolves this the same way in practice: download a JDK and use it to build your JDK. Debian documents the pain, and the Bootstrappable project has a whole page on it. They are fun reads, I highly recommend them. What makes JDK special is that there is no actively maintained JDK that can be built from source without a JDK. The authors had to go back quite a few years to find one, and bring it back to life.

“Unfortunately, most of the software needed for the bootstrap has been abandoned.” – Bootstrappable

Nixpkgs does the same. openjdk-25.0.4 is built by temurin-bin-25.0.3: a 135 MB prebuilt-tarball.

$ nix-store -qR $(nix eval --raw nixpkgs#openjdk25.drvPath) \
    | grep -iE 'temurin|jdk_x64'
/nix/store/…-OpenJDK25U-jdk_x64_linux_hotspot_25.0.3_9.tar.gz.drv
/nix/store/…-temurin-bin-25.0.3.drv

Nixpkgs makes it pretty easy to audit in the meta.sourceProvenance of the package.

$ nix eval --json nixpkgs#temurin-bin-25.meta.sourceProvenance
[{"isSource":false,"shortName":"binaryNativeCode"},
 {"isSource":false,"shortName":"binaryBytecode"}]

What does the source provenance of GuixPkgs’ openjdk-25 look like?

It’s a little whacky but the overall build chain in Guix is the following:

• jikes 1.22 (C++) compiles GNU Classpath 0.93, a free reimplementation of the Java class library.
• Classpath is enough to bring up JamVM 1.5.1, a JVM written in C. Now something can run Java.
• Now we can finally use Java tools: Ant and ecj, the Eclipse Compiler for Java.
• Then it doubles back and rebuilds all of that with the newer versions to get more modern JDK features.
• That stack finally compiles IcedTea 2.6.13 (OpenJDK 7) → IcedTea 3.19.0 (OpenJDK 8) → OpenJDK 9.
• After which it is one rung at a time: 9 builds 10, 10 builds 11, …, 24 builds 25.

Nineteen complete JDK builds, from a C++ program.

Analyzing the closure

We can use nix-store -q --graph to emit the entire closure as a dot file to visualize the differences.

I restyled the nodes and edges: dots instead of labelled boxes, and the red to highlight the derivations that exist only because of the JDK bootstrap.

The upper panel’s entire Java story is those two red dots and the one edge between them: temurin-bin-25.0.3 → openjdk-25.0.4. The lower panel’s is the 33-node constellation: the nineteen JDKs, plus the fourteen rungs of jikes, Classpath, JamVM, Ant and ecj that had to exist before anything could compile a JDK at all.

Why is the Nix graph still so big if I claimed it was from a binary distribution?

It turns out that a JDK closure is mostly not Java.

It is a large C++ program that wants X11, cups, fontconfig, freetype, alsa and zlib, sitting on a C toolchain that both sides bootstrap from source. That sub-graph is the same in both and it swamps everything.

I was curious to zoom-in and see the source-bootstrap portion of the derivation graph. Since that shared sub-graph is drowning the signal, I chose to filter it out.

Which derivations exist in this closure only because of how the JDK is bootstrapped? 🤔

That is a reachability query. We delete the Java-provenance nodes from the graph, see what is still reachable from the root, and whatever is left are the derivations that exist only because of the JDK bootstrap.

attributable = reachable(root) - reachable(root, without=java_nodes)

This lines up with our intuition. Nixpkgs only needs two derivations to bootstrap the JDK, while GuixPkgs needs 876.

Note Interestingly, the bootstrap build for JDK has rustc in its closure: IcedTea 8 wants GTK 2, which wants its own Mesa, which wants rust-bindgen 😱

The fact that Nix can now build OpenJDK from source is thanks to the amazing work done by Guix developers.

What started off as a fun art project, started during TacoSprint 2026, has now found relevance to those interested in bootstraping and reproducible builds.

We can then build a NixOS machine where every package is the Guix equivalent 😱.

Well, adeci¹ took the bait and we went even further than that. 😈

Say hello to Guix by Nix: a bootable VM where the kernel is Guix’s Linux-libre, the userland is translated Guix packages, and PID 1 is GNU Shepherd.

No systemd. No D-Bus. No NixOS activation. Not even a nix or guix binary in the guest. It’s Guile all the way down and Nix built all of it.

$ nix run --accept-flake-config github:adeci/guix-by-nix
...
Starting Guix-derived initrd
GUIX_BY_NIX_INITRD_MODULES_READY modules=virtio_pci,virtio_blk,...,overlay
GUIX_BY_NIX_INITRD_ROOT_READY device=/dev/vda
Starting Guix by Nix activation
GUIX_BY_NIX_ACTIVATION_READY
Starting Guix Shepherd as PID 1
SHEPHERD_NETWORK_READY address=10.0.2.15 gateway=10.0.2.2 resolver=10.0.2.3
SHEPHERD_PID1 parent=1 child=690

Welcome to Guix by Nix (experimental)
guix-by-nix login: root

Log in with root / guix-by-nix and poke around:

bash-5.2# ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:00 shepherd

bash-5.2# uname -r
6.12.62-gnu

bash-5.2# readlink -f "$(command -v ls)"
/nix/store/...-coreutils-9.1/bin/ls

As a reminder, even though these binaries live in /nix/store, they are not Nixpkgs packages. They were translated from Guix derivations using guix-transfer and built by the nix-daemon.

That coreutils was built from Guix’s package definition, source bootstrap and all , but it lives in /nix/store, because nix-daemon built it.

If you want to try out any of these packages on your own machine, you can use GuixPkgs.

$ nix run --accept-flake-config github:fzakaria/guixpkgs#coreutils

Tip Guix offers all packages built from source where Nix may offer it as a prebuilt binary. You can use GuixPkgs to get a source-built bootstrapped version of OpenJDK for example and all the whacky steps to get there.

How it works

How does this actually work? The project is a three-stage pipeline:

1. guix-transfer is the tool that translates Guix derivation graphs into Nix derivations.
2. GuixPkgs is the flake with all Guix packages, all built from the 357-byte seed, although there is a Cachix cache provided.
3. Guix by Nix assembles ~42 of those packages, a subset of the overall set, into a useable machine.

Warning AI was leveraged to write the initrd and activation scripts. That seems to trigger people lately, so consider yourself warned.

• The initrd is a custom shell script, interpreted by Guix Bash, that loads eight modules, mounts the root disk and the 9p store, and switch_roots.
• Activation (accounts, /etc, the setuid sudo copy) is another custom Nix-generated script run by Guix Bash.
• PID 1 is shepherd from Guix, with a small Scheme config that starts eudev, dhcpcd, openresolv, and a serial getty. herd status works like you’d expect.
• /etc/profile is compiled from Guix’s own search-path specifications, so GUILE_LOAD_PATH and friends point where Guix intended.

Every program that can be executed, every ELF file, every script interpreter, all traces back to a translated Guix derivation. The only things Nix authored are text files: the init scripts, the Shepherd config, /etc.

This is Guix by Nix.

Don’t trust me. Read the receipt.

“Every executable byte comes from Guix” is exactly the kind of claim that’s easy to say and easy to fudge. 🤥

A booted demo is great but that can’t prove it. A VM where ls secretly came from Nixpkgs boots identically.

The flake ships an audit-check that classifies every store path in the shipped closure by provenance, unpacks the compressed initrd², and inspects every executable payload. The audit derivation fails if anything is unclassified, any ELF file or script interpreter doesn’t trace to a translated Guix output, any /gnu/store reference survived translation, or anything systemd-shaped appears anywhere.

$ nix build --accept-flake-config github:adeci/guix-by-nix#status-report
$ jq '{rejected, claims}' result/status.json
{
  "rejected": {
    "wrapperPaths": 0,
    "nonGuixElf": 0,
    "externalScriptInterpreters": 0,
    "untranslatedGnuStoreReferences": 0,
    "unclassifiedPaths": 0,
    "systemdDbusLogindPaths": 0
  },
  "claims": {
    "translatedLinuxLibre": true,
    "wrapperFreeGuest": true,
    "allElfFromGuix": true,
    "allScriptInterpretersGuixDerived": true,
    "systemdFreeGuest": true,
    "fullyClassified": true,
    "emptyFirmwareBoundary": true
  }
}

The report includes a lot more information such as the exact Guix channel commit everything was translated from. There’s also some NixOS VM tests for good-measure. 🕵

Note The kernel is Guix’s linux-libre-lts, translated and built by nix-daemon like everything else. Nix wraps it in a thin adapter so NixOS’s VM tooling accepts it by augmenting with some additional metadata only; the bzImage is byte-for-byte Guix’s.

What’s next

One obvious next step would be a normal NixOS machine, where every package that exists in GuixPkgs shadows its Nixpkgs equivalent on $PATH. GuixPkgs offers such an overlay, but it needs quite a lot of CPU to build all of it….

A more realistic use case is to mix and match GuixPkgs and Nixpkgs packages in a single system. Get the best of both. I heard there were people in the Nix community who still want a non-systemd system, à la sixos. 🫠

For now, this project remains a minimal VM demo. If you make it PID 1 on real hardware, please send photos. 🙇

Having come from a background working on web-services, I was used to looking at individual time-series dashboards, sometimes over a few percentiles, and I was expecing to see some noticeable change but the data seemed too noisy to make any conclusions.

Turns out a colleague had also faced similar issues when trying to evaluate build-speed improvements. There are lots of variables that can affect the build: cold-cache, incremental, local, remote, etc. and the build times can vary wildly depending on the state of the system and the workload. She ended up leveraging a cummulative distribution function (CDF) to visualize the data and it was a revelation to me.

This led me to explore a few other different ways to visualize data, in addition to the CDF, and how a single image or statistic is often not enough to tell the whole story. This post will walk through a single synthetic dataset and show how different visualizations can tell different stories about the same data. The goal is to convince you to look at your data and not just summarize it with a single number.

Everything below comes from one synthetic dataset with a fixed seed. The full script can be found in this gist. It is a single file with a nix-shell shebang, so you can reproduce every figure exactly as long as you are using nix.

Note I leveraged AI to help generate the data and charts in this post for the story. If that bugs you, sorry. 🤷

The rollout that “made it worse”

Here is the setup: we operate a typical web-service and we rolled out a new caching tier over a week, hoping to cut request latency.

The change is fully deployed, and the latency dashboard that plots the mean looks like this:

Mean latency went up, from 112 ms to 122 ms. ☹️

A SEV is cut, we revert the change and write the postmortem. Right? 🤔

One number, four stories

It’s often good practice especially for web-services to look at various percentiles, especially the tail end of the distribution like the p95 and p99.

Now we have a problem, and the problem is that everyone is right. The mean says the change is a mild regression. The median (p50), says the change is a big-win, the typical request got nearly twice as fast. The p99 says it’s a SEV¹, the worst requests more than doubling.

The mean and the median, computed from the very same numbers, point in opposite directions.

Engineers are often taught to be data-oriented, but often it’s easy to cherry-pick the statistic that supports your argument.

Look at the shape

The next basic thing you can do with a distribution is plot its shape. Here are the two latency distributions, before and after, as densities:

There it is. 🤓☝️ The “before” is one tidy hump. The “after” is two humps.

This already explains the earlier contradiction but it’s a bit tricky to visualize correctly. The shape depends on a smoothing parameter we chose, the two fills muddy each other where they overlap, and it’s genuinely hard to read a percentile off it. I can see there are two populations; I can’t easily see where the median went.

The best chart you’re not using

The cumulative distribution function (CDF) answers one question for every percentile at once: what fraction of requests came in at or below x milliseconds?

CDFs are an extremely easy way to visualize multiple percentiles in a single chart. Depending on the curve, we can understand how the request latency is distributed across the entire population.

I found it incredibly useful to then plot the “before” and “after” CDFs on the same chart to see how they compare. You can then visualize the shifts at various percentiles and understand how the change affected the entire population.

In our story, the “after” curve has shifted left for request latencies below 140ms. That means more requests are finishing faster than before. The “after” curve is higher than the “before” curve to the right of 140ms which means more requests are finishing slower than before. The two curves cross at ~140ms which is the tipping point where the change goes from being a win to a loss.

Tip Two CDFs that cross are the unmistakable signature of a change that no single percentile can summarize, because the sign of the effect depends on which percentile you ask.

Who won, and by how much

The CDF tells us that the effect changes sign: faster or slower. The obvious next question is by how much, at each point in the distribution. We can plot for every percentile p, the after-latency minus the before-latency, known as the shift function.

Below the zero line the change is faster; above it, slower. We can visualize the magnitude of the change at each percentile.

The regression was there all along

So far we have looked at two frozen snapshots, before and after. Rollouts though are often not instant. In this story, we rolled out the new caching tier over a week, ramping from 0% to 100% of traffic.

What did each day look like? Stack one distribution per day and you get a ridgeline:

We can now visualize the regression emerging over time. We can see the main peak (the fast requests) sliding left as the rollout progresses, and a second peak (the slow requests) emerging on the right. The median is dropping, but the slow requests are quietly growing in number and latency.

The x-axis here is logarithmic. Latency is roughly lognormal, and on a linear axis the fast peak is a tall spike next to an invisible smear; the log axis is what lets both humps read as humps.

We can do something similar, squeezed into a single grid, as a heatmap. One column per day, colour for how much traffic lands at each latency:

You can sort-of make out a new population emerging faintly.

Any aggregate computed over the whole week would have blended these seven very different days into one muddy number and hidden the trend completely.

The bimodality had a cause

We’ve now thoroughly established what happened. The next question is why. This is in fact very similar to $DAYJOB where I had to cut the data by binary sizes (i.e. >50MiB) to observe the bimodality in the latency distribution.

In our story, new tier either serves a request from cache (a hit) or falls through to the backend with an extra hop (a miss). We can split the “after” requests by that property and draw a CDF for each:

Conditioned on cache outcome, each population is unimodal again.

We can easily see that cache hits are faster than the old baseline as they are shifted left.

Cache misses pay for the extra hop and land far to the right.

Why those requests?

“Some requests miss the cache” is a mechanism, but it isn’t yet a cause. Which requests miss, and why?

Each request carries one more field I haven’t used yet: its response size.

A cache holds small, hot objects; big ones get evicted or never fit. We can plot the latency against the response size, and colour each point by whether it was a cache hit or miss. We can also add a density to each margin to see how the two populations are distributed along each axis: a jointplot:

We can see two clean population clusters: small-and-fast (cache hits) and large-and-slow (cache misses). It’s clear that the bimodality in the latency distribution is caused by the bimodality in the response size distribution.

We now have something actionable: raise the cache’s max object size, or split the big responses. 🔥

A graph is worth a thousand numbers

Often a single panel or graph is too small to tell the whole story at best. At worst, it can be misleading. It is beneficial to have multiple views of the same data to understand the full story.

I was especially impressed with the way the CDF can convey the entire distribution in a single chart especially when comparing what might appear to be multiple populations.

“Do not trust any statistics you did not fake yourself.” – Winston Churchill

Large code model: The large code model makes no assumptions about addresses and sizes of sections. [cite]

In a previous post, I documented how I hit simple performance bottlenecks that made me believe that the code-model is largely theoretical in practice. In those cases, the fixes were evident and relatively small; however their omission was a hint at how no one uses the code-model because without them the performance penalty was a non-starter.

I hinted at some other ways I found the large code-model to still be lacking however I had not yet fully understood the pain of those failure modes. I left a small teaser near the bottom about Thread Local Storage (.tdata / .tbss) being one of the “fun failure modes”.

Turns out that it is worse than I thought. The instruction sequences the compiler emits for TLS are 32-bit by construction, and -mcmodel=large has nothing to swap them for. 🤦🏻‍♂️

To put that very bluntly, mcmodel=large is incapable of building complex large binaries despite its stated goal.

How do you even make a >2GiB binary?

The annoying part of this whole line of research is producing a test subject. Emitting 2GiB of real instructions into .text is genuinely painful, you’d have to generate and assemble billions of instructions, and the object file is enormous. Beyond that the cardinality of the problem space grows when you consider whether the code is position-independent, do we use a procedure-linkage-table (PLT), TLS, GOT and all the various nuances each compiler brings with how they layout and order sections in their default linker scripts.

But relocation overflow isn’t about how many bytes are on disk, it’s about virtual-address distance between a reference and its target. So we have a few options.

Uninitialized globals land in .bss, which is a NOBITS section: it occupies virtual address space but zero bytes on disk (the same sparse-file trick I wrote about in massively huge fake files). So a few thousand 1MiB arrays gives us gigabytes of address space essentially for free.

We can synthetically produce this pretty easily with the following script.

# N * 1-MiB globals, plus a function that touches each one
# so the compiler emits one relocation per array.
import sys
N, CHUNK = int(sys.argv[1]), 1024 * 1024
for i in range(N):
    print(f"char s{i}[{CHUNK}];")
print("long sum(void){ long a = 0;")
for i in range(N):
    print(f"  a += s{i}[0];")
print("  return a; }")
print("int main(void){ return (int)sum(); }")

$ python3 gen.py 4096 > huge.c
$ gcc -c huge.c -o huge.o
# 4 GiB of address space, 288 KiB on disk
$ du -h huge.o
288K    huge.o

Link it with the (default) small code-model and it falls over exactly where you’d expect, at the 2GiB signed-32-bit boundary:

$ gcc -no-pie huge.o -o huge
huge.c:(.text+0x780f): relocation truncated to fit: R_X86_64_PC32 against symbol `s2048' ...

s2048 sits at 2048 * 1MiB = precisely 2GiB.

Recompile the same source with -mcmodel=large and it links cleanly:

$ gcc -c -mcmodel=large huge.c -o huge_large.o
$ gcc -no-pie -mcmodel=large huge_large.o -o huge_large

The large model did its job: it replaced the 32-bit R_X86_64_PC32 references with 64-bit R_X86_64_GOTOFF64 sequences: a movabs of a 64-bit offset + an add.

The .bss trick is great for exercising the linker, but it’s a little unsatisfying and overly synthetic. I often want to mimic relocation failures as they traverse a large .text segment. We can leverage the assembler’s .fill directive to repeat instructions. We can generate a tiny source of N functions spaced 1MiB apart with a NOP sea between them, plus one dispatcher that calls every function. This requires each call to have a relocation and the calls to functions past the 2GiB mark overflow. 🔥

# N tiny functions spaced 1 MiB apart, plus a dispatcher that calls each one.
# .fill is a memset, so the assembler emits the bytes without codegen.
import sys

N, CHUNK = int(sys.argv[1]), 1024 * 1024   # e.g. 4096 -> ~4 GiB of .text

print(".text")
print(".globl main")
print("main:")
for i in range(N):
    print(f"  call f{i}")               # one relocation per function
print("  ret")

for i in range(N):
    print(f".globl f{i}")
    print(f"f{i}:")
    print("  ret")
    print(f"  .fill {CHUNK}, 1, 0x90")   # 1 MiB of real NOP bytes -> real .text

A quick primer on TLS access models

Now let’s do the exact same thing, but using __thread.

When you access a thread-local variable, the compiler doesn’t just load an address, it emits one of four access models, from fastest/least-flexible to slowest/most-flexible:

Local Exec (LE)

The variable lives in the main executable’s own TLS block. The offset from the thread pointer (%fs) is a link-time constant, baked directly into the instruction as a 32-bit immediate; uses R_X86_64_TPOFF32.

Initial Exec (IE)

The variable is in a module loaded at startup. The 64-bit thread-pointer offset is stored in a GOT slot, and the code loads it with a RIP-relative reference; uses R_X86_64_GOTTPOFF to reach the slot & R_X86_64_TPOFF64 to fill the slot.

General Dynamic (GD) / Local Dynamic (LD)

The fully general case for dlopen‘d modules, a call into __tls_get_addr; uses R_X86_64_TLSGD & R_X86_64_TLSLD.

Notice the pattern: the value that reaches the thread pointer can be 64-bit (i.e. it lives in a GOT slot), but every instruction that participates in a TLS access uses a 32-bit field. TPOFF32, GOTTPOFF, TLSGD, TLSLD are all 32-bit.

# Same as gen.py, but every array is thread-local (__thread) so it lands in
# .tbss and each access emits a TLS relocation instead of a plain data one.
import sys

N, CHUNK = int(sys.argv[1]), 1024 * 1024   # e.g. 4096 -> ~4 GiB of .tbss

for i in range(N):
    print(f"__thread char t{i}[{CHUNK}];")

print("long sum(void){ long a = 0;")
for i in range(N):
    print(f"  a += t{i}[0];")
print("  return a; }")
print("int main(void){ return (int)sum(); }")

$ python3 gen_tls.py 4096 > tls_huge.c   # like gen.py but each array is `__thread`
$ gcc -c -mcmodel=large tls_huge.c -o tls_huge.o
$ du -h tls_huge.o
288K    tls_huge.o

4GiB of .tbss, still tiny on disk. Now look at the relocations the large code-model generated:

$ readelf -r tls_huge.o | awk '{print $3}' | grep R_X86 | sort | uniq -c
      1 R_X86_64_GOTOFF64
      2 R_X86_64_GOTPC64
      2 R_X86_64_PC32
   4096 R_X86_64_TPOFF32

Every single TLS access is R_X86_64_TPOFF32 which is 32-bit, even under -mcmodel=large. 🫣

$ gcc -no-pie -mcmodel=large tls_huge.o -o tls_huge
tls_huge.c:(.text+0x25): relocation truncated to fit: R_X86_64_TPOFF32 against symbol `t0' defined in .tbss ...
tls_huge.c:(.text+0x36): relocation truncated to fit: R_X86_64_TPOFF32 against symbol `t1' ...

The exact same large code-model that just happily linked 4GiB of ordinary .bss fails on 4GiB of .tbss.

This isn’t a GNU quirk either. LLVM (clang & lld) does exactly the same thing, albeit with lld’s diagnostic being a bit friendly by printing the actual offset and the window it has to fit in:

$ clang -c -mcmodel=large tls_huge.c -o tls_huge.o
$ clang -no-pie -fuse-ld=lld -mcmodel=large tls_huge.o -o tls_huge
ld.lld: error: tls_huge.o:(function sum: .ltext+0x17): relocation R_X86_64_TPOFF32
  out of range: -4294967296 is not in [-2147483648, 2147483647]; references 't0'

-4294967296 is exactly -4GiB, t0 sitting at the far bottom of the TLS block, and [-2147483648, 2147483647] is the signed 32-bit window TPOFF32 has to live in.

lld even parks large-model code in a .ltext section.

Why the large model is powerless here

I was a little confused at first: R_X86_64_TPOFF64 exists. Why doesn’t the compiler use it, especially with mcmodel=large ?

A relocation type is glued to the specific field it patches and is determined by the code-sequence emitted by the compiler. Here’s the local-exec sequence -mcmodel=large emits:

   4: 64 48 8b 04 25 00 00   mov    %fs:0x0,%rax
   d: 48 05 00 00 00 00      add    $0x0,%rax
                             R_X86_64_TPOFF32  x
  13: 8b 00                  mov    (%rax),%eax

The offset is applied with add $imm32, %rax. The x86-64 encoding for that instruction has only a 32-bit immediate field. There is no 64-bit form of it. The only relocation that can physically patch that field is a 32-bit one: TPOFF32.

So where is TPOFF64 used? It’s not in the code (.text) at all. It lives as a dynamic relocation on the GOT slot used by initial-exec. The code side only holds the 32-bit GOTTPOFF that points at the slot:

   4: 48 8b 0d 00 00 00 00   mov    0x0(%rip),%rcx
                             R_X86_64_GOTTPOFF  y
   b: 64 48 8b 04 25 00 00   mov    %fs:0x0,%rax
  14: 48 01 c8               add    %rcx,%rax
  17: 8b 00                  mov    (%rax),%eax

TPOFF64 patches the 8-byte (64bit) data word in the GOT. The instruction that reaches that word (GOTTPOFF) is a 32-bit PC-relative relocation. So even initial-exec has a 32-bit link in the chain: the GOT slot must sit within 2GiB of the code. 😭

It fails for exactly the mode it’s meant for

local-exec is the mode you want for a big statically-linked executable’s own thread-locals. It’s the fast path: the thread-pointer offset is a link-time constant, so the access is a direct add $imm32 with no memory load, no GOT, no indirection. Unfortunately, TPOFF32 is limited to 32bits and overflows.

I would argue it’s also the mode most likely to need a large binary, since it’s what a giant static executable uses for its own TLS.

What if we fall back to a slower mode? Unfortunately, each one of them has a 32-bit field too:

• initial-exec GOTTPOFF is 32-bit RIP-relative reach to the GOT slot.
• general-dynamic TLSGD is 32-bit RIP-relative.

The use of GOTTPOFF seems especially confusing. The large code-model already reaches the GOT at 64 bits everywhere else: GOTPC64 for the base, GOT64 for the slot, so the omission seems surprising. The 64-bit machinery is right there. TLS just doesn’t use it. 🥲

This is a hole in the ABI, not a compiler bug

After having dug into it, it’s not simply a bug in GCC or LLVM. Look again at what clang emits for a single local-exec access under -mcmodel=large:

   4: movq %fs:0x0, %rax
   d: addq $0x0, %rax           # R_X86_64_TPOFF32 x
  13: movl (%rax), %eax

It’s a plain addq $imm32, %rax. There is nothing stopping clang from emitting a 64-bit form instead:

movabs $x@tpoff, %rdx      # a TPOFF64 in a movabs immediate
addq   %fs:0, %rdx

TPOFF64 could patch that immediate, exactly the way GOTOFF64 already patches a movabs immediate for ordinary data.

Oddly, the relocation type is not the missing piece. What’s missing is a code sequence in the x86-64 psABI that uses a 64-bit TLS offset as an instruction immediate. The ABI simply never defined a large code-model TLS access model, so TPOFF64/DTPOFF64 only ever appear in GOT slots, never in code, and no toolchain can emit what the spec doesn’t describe.

The large code-model “exists”, but it doesn’t actually deliver arbitrarily-large binaries. For thread-local storage it can’t, because the specification never defined how to.

This is one more reason I’m working toward making massive binaries possible. If you want to follow along, the discussion is over in the x86-64-abi google-group where I’ve posted an RFC, and we have started an LLVM Massive Binaries working group and hold monthly meetings.

I am sure this will put people off from reading my blog just stating this, but I decided to take a stab at some design changes with AI and some minor site improvements that I have always wanted to do.

If you think the site looks much worse, please let me know.

Some small improvements and quality-of-life enhacements:

• A small /nix/store entry in the footer to showcase the site is built with Nix.
• A Jekyll plugin that automatically creates source-sets for the images I use in my blog posts.
• A Jekyll plugin to create a short content hash for the stylesheets to avoid the browsers caching stale CSS.
• A Jekyll plugin that generates a little curved line for the masthead that is the sha256 of the page.
• Redesigned the layout to allow side-bar content.

Look at me! I’m sidebar content I will undoubtedly abuse.

I continued to try and give it my personal flair to avoid it looking like AI-slop but 🤷

I enjoy these lofty goals to push Nix and the surrounding ecosystem forward. I am bold if not stupid.

I left the last earlier post with one potential idea of how to get there:

We could patch the Linux kernel so that $ORIGIN is supported in PT_INTERP and the shebang.

I waded through the complexity of sending patches over email (turns out I actually enjoy this workflow!), and sent a proposal to the Linux kernel mailing list.

My first attempt here proposed simply adding direct support for $ORIGIN in the Virtual File System (VFS) subsystem.

I waited nervously. I was expecting the result from what I had come to read about online; someone non-politely telling me to F$#CK OFF because there is something I missed, misunderstood or did not consider. 🤬

The result was completely different. 😲

Christian Brauner, the maintainer for VFS responded to me in good faith, asking for the rationale for the change and eventually proposing some ways in which such a support could make it into the subsystem.

Note It definitely helped having someone like John Ericson chime in and advocate why having a non-fixed interpreter (PT_INTERP) is useful to Nix and other use-cases (i.e. Buck & Bazel).

He offered that potentially we could leverage eBPF as a programmable way to select an interpreter through binfmt_misc.

Whoa! 🤯

I wanted to merely allow $ORIGIN but a programmable selection could let us do anything!

The idea must have really intrigued him because soon-after, on his vacation, Christian offered the first draft of such a solution. We went back and forth a little over the mailing list and the end result is a patch series that will make its way into -next branch in the near future.

If you don’t know what eBPF is or binfmt_misc, WTF did we just collaborate on?

Let’s take a look!

I won’t do eBPF justice, and there are plenty of articles online about it as it’s quite in-vogue at the moment.

tl;dr; You can write programs in a C subset that gets compiled to an instruction set whose virtual machine is running within the kernel. Shouldn’t the kernel be super fast? Yes, the programs are jitted to their native CPU architecture and the programs have a fixed-time slice. Isn’t this some crazy vulnerability for the kernel? Before any code is loaded it is “verified” to be safe. Checkout this guide for more info.

We can now support $ORIGIN with a relatively simple eBPF program:

SEC("struct_ops.s/match")
bool BPF_PROG(nix_match, struct linux_binprm *bprm)
{
  return !bpf_strncmp(bprm->buf, 4, "\x7f" "ELF");
}

SEC("struct_ops.s/load")
int BPF_PROG(nix_load, struct linux_binprm *bprm)
{
  char path[256];
  long n;

  n = bpf_path_d_path(&bprm->file->f_path, path, sizeof(path));
  if (n < 0)
    return n;

  /* derive the loader location from the binary's path */

  return bpf_binprm_set_interp(bprm, path, sizeof(path));
}

SEC(".struct_ops.link")
struct binfmt_misc_ops nix = {
  .match = (void *)nix_match,
  .load = (void *)nix_load,
  .name = "nix",
};

Once the above program is loaded and registered into the kernel, we then ask the binfmt_misc subsystem to trigger it. Checkout this thread if you want to see the complete example.

> bpftool struct_ops register nix_origin.bpf.o /sys/fs/bpf
> echo ':origin:B::::nix:' > /proc/sys/fs/binfmt_misc/register

What does that mean?

It means that every binary now triggers the nix_match function above, in this case any ELF file, but it could be executables with a new segment like PT_INTERP_NIX, and the kernel will ask nix_load to determine the interpreter to use dynamically.

Our special BPF program has support for $ORIGIN 💥

What else could you do?

Well we can now even completely replace the traditional QEMU binfmt_misc registration script with a BPF program now like this one.

What else can we do?

Since we can now programmatically select our interpreter based on anything in the file, we can do quite a lot. I’m keen to hear your suggestions and ideas 💡.

Some of the smaller items are that we can even support $ORIGIN in the shebangs (#!$ORIGIN/bin/ld.so) very easily as seen here: we simply look at the first 256 bytes of the file and look for $ORIGIN to trigger.

One downside or side-effect of the traditional binfmt_misc hand-off was that the way in which the desired final binary was invoked was non-transparent.

The registered interpreter becomes the process. It owns the entire process identity, and the binary you actually asked to run gets demoted to an argument. For wine or qemu that’s acceptable as they are emulators but for a per-binary BPF loader that might pick a traditional ld.so it does not make much sense.

This leaks in a few painful ways but the simplest are :

• argv[0] and /proc/<pid>/cmdline show the interpreter invocation, not what you executed.
• /proc/self/exe names the interpreter. Relocatable programs commonly locate themselves through /proc/self/exe, and instead they find the dynamic linker. 😩

Christian sent a large patch series for this as well. His latest patch series adds two new dispatch modes that close the gap from opposite ends and covers a few other gotchas that these modes can fix.

The loader substitition L is the one I’m most excited about for Nix.

With the L flag, the kernel executes the matched binary natively as the main image, and merely substitutes the registered interpreter for the loader named in the binary’s PT_INTERP. binfmt_misc stops being a hand-off and becomes a plain PT_INTERP override. There’s no contract and no identity to reconstruct, so a stock dynamic loader works unchanged.

Where does this leaves us?

I’ll be tracking the Linux kernel releases and, once this lands in -next and ships in a tagged release, I plan to upstream a NixOS module that registers the $ORIGIN support at boot. 🎉

The plan is to gate it on a new PT_INTERP_NIX segment rather than matching every ELF file. That keeps things backwards compatible: the BPF handler only kicks in for binaries that explicitly opt-in by carrying the new segment. This means Nix produced binaries continue to work without the BFP handler but those that have it may elevate themselves to relocatable status.

A ship in harbor is safe, but that is not what ships are built for. — John A. Shedd

Warning If you are pissed off reading this blog post, I guess, mission accomplished ? Try not to take life too seriously. Unfortunately, there’s a lot worse things in life than my opinions on Nix.

Seems like it’s all too easy to get people in Nix flustered, angry and out with their pitchforks. All it takes is someone proposing a markdown file for the community to lose its mind.

Having been a member, exposed to and part of the Nix/NixOS community for many years, I thought I would share some personal opinions, some which are self-evident and others that are purely philosophical.

Nix is brilliant and deeply flawed. Both things are true at the same time. The documentation is infamously terrible (although getting better!), the language is foreign to read for many and there are still growing pains from the governance changes. You should be free to say all of this and yet still believe Nix to be the best idea in the industry, at least this decade.

The emperor has no clothes.

Nix isn’t for everyone.

“If we just fix the documentation and the onboarding, then there’s no stopping Nix & NixOS” - Most Nix users

Nix is “having a moment”. Growth on any measurable metric is growing. There are endless posts about it.

Part of human nature is the desire to win. As a result, there is a persistent fantasy that if we just fix the documentation, just smooth the onboarding, just ship a friendlier installer and so on, then everyone, even your grandmother, can be using NixOS.

Nix is not meant for everyone.

Nix is a power tool. Power tools have a learning curve and can occasionally take a finger. I continue to use a non-NixOS Linux machine in addition and guess what, it works well enough. It is surprisingly stable and despite what Nix leads us to believe, not everything is on fire.

Our obsession with mass adoption has warped priorities and diluted the amazing possibilities Nix could allow by having to make it more palatable for broader appeal.

Being American is perceived to be a problem.

Nix originated in Europe. It should be no surprise that it skews heavily European. We can easily affirm this from the 2025 community survey.

Europeans are different from Americans. We hold different cultural values and priorities. Both have traits that I wish the other emulated. Unfortunately, where they clash however is often a point of contention.

Americans are capitalist maximalists. The idea of the “American dream” is tied to it. We also have the largest military budget in the world. You’re rarely more than a degree or two from a through-line between a business and the military. This clashes with much of the European worldview.

As a result, there’s a bit of an undercurrent where being an American or an American corporation is problematic in the community. Your position is suspect from the start, assumed to carry ulterior motives and it curdles into a purity test.

AI is useful.

I am clearly in the “AI is useful” camp. I have written before about how much LLMs have unlocked for me personally.

The Nix ecosystem is probably the best poised for the AI-wave. I have found a newfound joy and love for my NixOS machine now with LLMs. All those weird quirks that bugged me I have been able to resolve, and declaratively reproduce for future generations. One-off AI written tools can be written and stored in my NixOS configuration with a sense of assurance that they will not collide or interfere with the rest of the system.

Unfortunately there’s a loud contingent that treats AI output at best as technically unsound and at worst as some moral failing of the user. Despite nixpkgs offering AI tools, an AGENTS.md file was seen as heresy.

I particularly enjoyed Linus Torvalds, on a kernel mailing list articulating better than me Linux’s position on AI:

AI is a tool, just like other tools we use. And it’s clearly a useful one. […] Anybody who doubts that clearly hasn’t actually used it.

These are tools we can use to push Nix & nixpkgs further.

BDFL is undervalued.

The democratic process is great for society but a software project originated from someone. Someone had the vision, birthed the idea, worked tirelessly on it and then attracted others to contribute to their ongoing vision. In the case of Nix, Eelco created Nix in 2003 as part of his PhD research. The NixOS foundation didn’t exist until 2015. That is over a decade of work towards a project driven by his own vision as the Benevolent Dictator for Life (BDFL).

A non-democratic model works especially well in open-source because you are free to fork the software and try your hand at your own ideas if you disagree – the same cannot be said for our shared geography.

A clear vision, whether you agree with it or not, is refreshing. Someone who can say yes or no and is not simply stonewalled by design-by-committee. DHH had said it poignantly well: “Using open source software does not entitle you to a vote on the direction of the project.” [cite].

Abandon macOS and definitely Windows.

Every hour spent making Nix pleasant on macOS is an hour not spent making Nix extraordinary on Linux. You would never catch an iOS developer working on Linux and yet it pains me to see those who target the Linux platform working on a Mac.

The single most-impactful advice I give to those earlier in their career is to use a Linux machine, especially if you build and write software for Linux (i.e. web stuff). There are rarely any tips to fast-track your career beyond time except for this one.

For Nix, Darwin support is a bottomless tax. Closed toolchains, an SDK that shifts under you every release, a sandbox that fights you. All to chase an OS whose entire philosophy (opaque, proprietary, convention over purity) is the antithesis of Nix.

When we have to target solutions that cover wildly different platforms, the end result is muddled and limited. Beauty, elegance and innovation emerge when you apply constraints and restrict a problem.

Flakes are meh.

Flakes are here to stay and yet their adoption is constantly brought up in order to validate its existence. I default to using it for my new projects, mostly because it’s easier at this point and it seems annoyingly tied to the new CLI format.

Upon reflection though, I don’t feel like I have gained anything really over npins or niv. Subjectively my nix evaluations feel slower as now I’m fetching many more nixpkgs or jump through hoops to make every flake follow each other defeating the whole purpose of separate trees.

single-user install is great.

Unless you are sharing a laptop with your family or are using a mainframe from 1980, you won’t have more than a single user on your machine. Despite this, the default installation we guide users towards is one designed for multiple users.

The multiple-user install adds unnecessary complexity many don’t need or won’t understand: a build daemon, pool of nixbld users, systemd service, etc…

In contrast, the single-user install is radically simpler to run, operate and triage. I don’t have to remember whether I am setting configuration for my “client” or the “daemon” and that there is a difference.

Multi-user is the right default for a shared build farm. It is overkill for the single human it’s actually installed on the majority of the time.

Think bigger.

Nix is the closest thing we have to a solution where we can rebuild the entire world reproducibly. It is used by the Software Heritage Foundation as a way to reliably collect, preserve, and share all software that is publicly available in source code form [cite].

Some of the most amazing technology that exists in this world exists when one can make changes at multiple layers throughout the stack. This is the secret sauce to many of the hyperscalers of today. This is table-stakes for NixOS. You can implement a solution that requires new: application, compiler, library, runtime and kernel all within a single commit.

Despite the ability to wildly diverge from traditional distributions, innovate and differentiate, we largely replicate the status-quo – albeit more reproducible. Whether it’s constraints imposed by needing to accommodate alternate platforms (e.g. macOS) or fearing alienating more novice-users we limit the potential of what Nix could do.

Are you pissed off? Let us still be friends.

Of course I’m leveraging an LLM to help me do this! To pre-seed the context of the LLM, I asked it to read the https://lore.kernel.org/ thread.

Uh oh. Looks like they have adopted Anubis, which is an HTTP proxy that requires proof-of-work before allowing access to the resource.

Did this really do anything?

Unfortunately, no.

My AI diligently came up with anubis-fetch, which you can find at https://github.com/fzakaria/anubis-fetch. The tool tries to natively solve the proof of work or, as a last resort, will launch Chromium to visit the URL.

This tool also impersonates a real Chrome TLS/JA3 fingerprint natively via req so it clears passive Cloudflare blocking too. ☝️

# HTML to stdout
$ anubis-fetch https://lore.kernel.org/linux-mm/some-thread/T/

# readable plain text
$ anubis-fetch --text https://lore.kernel.org/linux-mm/some-thread/T/

So who did we stop?

The exact adversary Anubis targets defeats it trivially.

The whole use of Anubis feels regressive and marginalizes those without access to “good” AI.

For a scraper, solving the Anubis challenge is a one-time, amortized-to-zero cost since the cookie can be cached and reused. For a human, it’s seconds of spinner, battery drain on every fresh visit. They can’t amortize anything amongst each other.

This “regressive tax” is paid even more so by those with weaker devices or who access the content on their phone. Clients that don’t leverage JavaScript (e.g., text browsers (w3m/lynx), screen readers, RSS readers) are completely left out.

Did deploying Anubis stop any of the aforementioned bot-farms or are they mildly inconvenienced when they had to augment their bots to support a new proof of work solution briefly?

The irony is that Anubis’s goal is to stop AI but it was incredibly easy for AI to circumvent it and yet the cost to humans and an open web remains.

With the presumption Anubis is now a regressive tax, how much does it cost us?

Every number here is a rough estimate. This is not a environmental argument at all since the bot-farmers and AI tools themselves are using many orders of magnitude more energy. Nevertheless, it’s interesting to see how much time is spent doing proof-of-work challenges that marginalize people.

Difficulty d is the number of leading zero hex characters the hash must have, so the expected work per solve is W = 16^d hashes.

Difficulty 4 is the common default. Rates assumed: ~50 MH/s native (Go), ~0.5 MH/s in-browser JS; “felt” wall-clock includes page load, the worker, and the reload.

Let C be the number of Anubis challenge-solves per day, worldwide. Assume a felt time of t = 2 s and device energy E = 20 J per solve (screen + CPU).

• Human-time / year = C × t × 365 / 3.15×10⁷
• Energy / year (kWh) = C × E × 365 / 3.6×10⁶

Collectively we are wasting an impressive amount of time waiting for access to websites; time we didn’t spend before the AI era. As a human, time is precious and finite to me, whereas to a robot it is not.