Upon further reflection, I realized that the intensional model is an even more substantial common denominator. What is the intensional model?

TIP I’ve written before about Nix’s intensional model, better known as content-addressed (CA) derivations. It has been “experimental” in Nix for years. 🧪

In the intensional model, outputs are addressed by their content, and actions refer to their inputs by content as well. The extensional model instead addresses an output by the recipe and inputs that produced it. Nix also calls this “content-addressed derivations” (CA derivations). Bazel and buck2 don’t name their model, but they use content-addressing as well. Consider this coincidence: Eelco’s PhD thesis was published in 2006, the same year that Google started development on Blaze, the predecessor of Bazel [cite].

Let’s take a closer look at the three systems with a simple C example and see how all three implement the intensional model. 🥸

// greet.h
void greet(const char *name);

// greet.c
#include <stdio.h>
#include "greet.h"

/* say hi */
void greet(const char *name) {
  printf("hello, %s\n", name);
}

// main.c
#include "greet.h"

int main(void) {
  greet("world");
  return 0;
}

In the traditional extensional model (input-addressed), a change anywhere upstream of a build action invalidates the action’s output. This occurs even if the change is irrelevant to the output, such as a comment change in a source file. In contrast, the intensional model (content-addressed) only invalidates an action’s output if the content of the output itself changes.

In our C example, if we were to change the comment in greet.c from say hi to say hello, the compiled object file greet.o would still be byte-for-byte identical, as the comment does not affect the compiled output. In the ideal world, the linking of hello would not need to be redone, as its inputs have not changed in content.

{ pkgs ? import <nixpkgs> {}, ca ? false }:
let
  extra = pkgs.lib.optionalAttrs ca { __contentAddressed = true; };
in rec {
  # The `cp` is needed otherwise the source file's store path ends up
  # in the object file, which is a leak of the extensional model.
  greet = pkgs.runCommandCC "greet" extra ''
    cp ${./greet.h} greet.h; cp ${./greet.c} greet.c
    mkdir -p $out
    cc -c greet.c -o $out/greet.o
  '';
  hello = pkgs.runCommandCC "hello" extra ''
    cp ${./greet.h} greet.h; cp ${./main.c} main.c
    mkdir -p $out/bin
    cc main.c ${greet}/greet.o -o $out/bin/hello
  '';
}

By default, Nix uses the extensional (input-addressed) model. If we edit the comment and rebuild:

$ nix-build -A hello
building '/nix/store/3hwrlpnji5bh5bajdg52s7zbg7ihs6j6-greet.drv'...
building '/nix/store/2k5is7yim1j8pb3qjz89bw45cqzcy2y5-hello.drv'...
/nix/store/122jc3iz6qkrgm907kdz7rrk3lp4qrma-hello

Everything rebuilt. 😩

The store path in the extensional model is a hash of the recipe, and the recipe for hello includes the recipe for greet. Recursion is fun! Change one byte anywhere upstream and every path downstream changes with it. 🌀

Warning The two hello binaries are not even byte-identical: the cc-wrapper puts $out/lib in the RUNPATH, so the binary contains its own store path. The extensional model leaks into the bytes. 🙃

If we enable the build to use content-addressed derivations we observe greet rebuilt, but hello did not. 🎉

$ nix-build --extra-experimental-features ca-derivations -A hello --arg ca true
building '/nix/store/qpa1q7g5yjvhdbnfap4y3r2glzyflyv4-greet.drv'...
resolved derivation: '/nix/store/1xnbxrrprs50879ivzmjpv7cbk44w258-hello.drv' -> '/nix/store/rs2j6r2gvz2vi7nfvqj1s3yxx4lqpa3m-hello.drv'...
/nix/store/l2nrycgcg97vrz4nb59icwz9i04ys8jg-hello

You can read demystifying Nix’s intensional model for a more detailed explanation of how CA derivations work, but the short version is that Nix rewrites hello.drv to point at greet’s content-addressed output instead of greet.drv. The resolved rs2j6r…-hello.drv is the same one we built before the edit.

Since no “input” changed, Nix avoids the rebuild.

This is the same behavior as Bazel and buck2, which are intensional by default.

cc_library(
    name = "greet",
    srcs = ["greet.c"],
    hdrs = ["greet.h"],
)

cc_binary(
    name = "hello",
    srcs = ["main.c"],
    deps = [":greet"],
)

If we edit the comment and rebuild with Bazel or buck2, we see that the compile ran, but the link was an action cache hit.

$ bazel build //:hello --explain=explain.log
INFO: 2 processes: 1 action cache hit, 1 internal, 1 linux-sandbox.

$ cat explain.log
Executing action 'Compiling greet.c': action changed since cached execution.

$ buck2 build //:hello
Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD SUCCEEDED

$ buck2 log what-ran
build  root//:greet (c_compile greet.c (pic))  local  ... clang -c ./greet.c ...

Bazel and buck2 do not have an input-addressed mode. They always use the intensional model. An action’s key is a hash of its command line (to force rebuilds) and the content digests of its inputs.

A change only travels as far as the bytes actually change. This optimization is called early cutoff, and it is the whole selling point of the intensional model.

There is a clear mapping between Nix’s CA derivations and Bazel’s action cache.

A resolved derivation is a Bazel/buck2 action. 🤯

You may see the output paths in Bazel and buck2 and think they are extensional (i.e. bazel-out/k8-fastbuild/bin/hello), but they are just a convenient place on disk. The real addresses are the digests. buck2 can even make this explicit with declare_output(has_content_based_path=True).

Until dynamic derivations stabilize, people will continue to reach for Bazel and buck2 for the granularity on large builds. The intensional model is a big part of why they are so fast, and Nix has the same model. The two implementations are nearly identical in concept.

Nix described the intensional model in 2006. Google started building on it the same year. Twenty years later, maybe it’s time they compared notes. 🥸

What if you could travel back in time to see what happened? What if you could see the exact moment a value was written, and who wrote it? What if you could run your debugger backwards?

Rewind is a deterministic virtual machine that records a program’s execution once, and then lets you replay it identically, many times. This lets you run the debugger backwards, and see the history of a value in memory, including who last read and wrote it.

Turns out that you can go backwards by fast-forwarding.

Going backwards changes how you debug: you can see how a value got there, not just what it is now. In who wrote this byte? rewind blame found the program that wrote a byte of a file. That was the easy case. Every write to a file goes through the kernel, and Rewind’s kernel logs each one. Writes to memory don’t go through the kernel, so there is no log to read.

So instead we go backwards, by running forwards again. This is how rewind blame can tell us who last read and wrote a value in memory, and it is the same trick that lets gdb run backwards as well. Let’s see how it works.

Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM?

Two tellers, one account

Let’s use the bank example from Nix wrote half of my debugger. Two tellers each deposit 10, twenty times, into one account:

static long balance;

static void deposit(int teller, long amount)
{
	long seen = balance;
	char line[64];
	int n = snprintf(line, sizeof line, "teller %d: %ld + %ld\n", teller, seen, amount);

	if (write(1, line, n) == n) {
		balance = seen + amount;
	}
}

In our example, crucially, there is no lock around the read and write of balance. If the other teller runs in between, one of the deposits is lost.

rewind check runs the program under different thread schedules until one fails:

$ rewind check --epoch 1790985600 github:fzakaria/rewindvm#bank
schedule   0: exited:0             4547 steps  2ffd6b62eb30  run b1d6981401d5d15f  holds
schedule   1: exited:2             3768 steps    run 36036ed36994d7d8  violates same-outcome, sdk
...
failing: run 240115d5505149aa, schedule 1 over steps 3245..3246
failing run violates sdk: step 3292: always "balance-adds-up" failed: balance 310, expected 400

check narrows schedule 1’s failure down to steps 3,245 and 3,246 and records that run, 240115d5, which is the one we will dig into. We expected 400 and got 310. Nine deposits are gone. 😬

rewind blame –at

Our program prints the balance at step 3,291. We can run rewind blame at that step and the variable² to see who last wrote it:

$ rewind blame 240115d5 --at 3291 balance
balance = 310   8 bytes at 0x56055a0ef030, thread 139, bank.c:69

last written at step 3282 by thread 141 (bank)
  bank.c:41  teller  balance = seen + amount;
  was 300

other threads just after the write, in step 3282:
  thread 139  bank.c:65  main  __GI___clone3

searched back to step 3210, 1 keyframe window, 1 write, 3.1s

Rewind goes “backwards” through the run, looking for the last write to that value. We can see that the last write was at step 3,282, by teller 2 (thread 141) on line 41.

But wait, 300 + 10 is 310. That write is fine! 🤔

The last write is rarely the bug. The money went missing earlier, when a teller wrote a value it had read much earlier. We need to augment blame with the reads too, to see that.

Who read it?

--reads lists every read and write of the value, back to the start of the process. Here are the rows around where it goes wrong:

$ rewind blame 240115d5 --at 3291 balance --reads
  step  thread           access  value               where
...
  3244  140 bank         read    110                 bank.c:36  teller  long seen = balance;  balance
  3248  141 bank         read    110                 bank.c:36  teller  long seen = balance;  balance
  3250  140 bank         write   120                 bank.c:41  teller  balance = seen + amount;  balance
  3250  140 bank         read    120                 bank.c:36  teller  long seen = balance;  balance
...
  3257  140 bank         write   190                 bank.c:41  teller  balance = seen + amount;  balance
  3257  140 bank         read    190                 bank.c:36  teller  long seen = balance;  balance
  3258  140 bank         write   200                 bank.c:41  teller  balance = seen + amount;  balance
  3263  141 bank         write   120                 bank.c:41  teller  balance = seen + amount;  balance
...
searched back to step 3210, 1 keyframe window, 40 writes, 41 reads, 1.8s

Reading the chronology, we can see that teller 2 read the balance at step 3,248, and then teller 1 made nine deposits, taking the balance from 110 to 200. Then at step 3,263 teller 2 wrote its stale value of 120, overwriting the balance and losing the nine deposits (90). 💣

In this example, the bug is obvious, but in a real program it can be hard to find. The --reads option shows you the whole history of a value, and you can often see where it went wrong without ever setting a breakpoint. You can even see the bug in the plot of the balance over time:

The example is simple and the fix is obvious: a lock around the read and write. In a real program, the bug can be subtle and the fix complex. Seeing the history of a value, and who last wrote it, can be invaluable in tracking down the root cause.

In the app

The Rewind app gives the same information but in a more interactive way. The source panel underlines every value that blame can watch:

You can click on the value to see who wrote it, and then click Show reads too to see the whole history of that value.

Show reads too draws every access, one lane per thread. Teller 2’s lone read sits in the middle of teller 1’s deposits, and that one dot is the bug. 🙌

gdb, backwards

If you would rather drive, rewind gdb can go backwards too by setting a watchpoint and reverse-continue. Here it is forked at step 3,264, just after teller 2’s stale write:

$ rewind gdb 240115d5 3264
(gdb) watch balance
Hardware watchpoint 1: balance
(gdb) reverse-continue

Thread 3 hit Hardware watchpoint 1: balance

Old value = 120
New value = 200
0x000056055a0ec50d in deposit (teller=2, amount=10) at bank.c:41
41			balance = seen + amount;

You can navigate the stack, inspect variables, and step backwards with reverse-step and reverse-next.

Rewinding by fast-forwarding

How does this all work? How do you run a VM backwards?

You don’t. 😅

Rewind replays are completely deterministic: the same run produces the same result every time. Turns out that a simpler way to go backwards is to run forwards again, but stop one instruction sooner than last time. That is all blame and reverse-continue do.

Rewind keeps a checkpoint of the machine at the start of a step. It rewinds to that checkpoint and then runs forward back to the step you want to inspect making note of every read and write along the way. If no reads or writes are found, it rewinds to the previous checkpoint and runs forward again. It keeps doing this until it finds the last read or write before the step you want to inspect.³

Deterministic replay is what makes this work, and it keeps the implementation simple: going back is just going forward again from earlier.

I’ve been having a lot of fun debugging my own programs this way. It’s a bit like time travel without the paradoxes. You can see the whole history of a value, and who wrote it, without setting a breakpoint and restarting the program.

It’s git blame for your program’s memory. 🔍

Let’s start with a simple story. When I worked at AWS on RDS (Relational Database Service), we built the control plane for the service using workflows, of course using SWF (Simple Workflow). The workflows would execute a series of steps to create, modify or delete a database instance. Every so often we’d see something that looked impossible: a workflow that seemed to run its steps out of order.

What we actually observed was a pause. A worker would pick up a step and then stop for an unexpectedly long time. While paused it stopped heartbeating, so SWF decided it was dead and reassigned the step to another worker. The new worker finished the step and the workflow moved on. Then the first worker woke up. As far as it could tell no time had passed, so it carried on with the step it was holding, causing havoc in its wake.

We suddenly realized that we were not adequately testing all the possible schedules of events. The workflow code was correct, but the schedule that triggered the bug was not one we had ever seen in our tests. It was a rare schedule, but it was not a rare bug. So we began adding idempotency tokens to every step and making sure that every step could be run multiple times without breaking anything. It was a lot of work, but it made the system much more robust.

What’s the purpose of this story?

I am building Rewind VM, a deterministic VM. It helps find bugs in software primarily by running the same code under many different schedules.¹ The mere act of shuffling the schedule has found many bugs thus far.

The software under test is not even a distributed system, such as the workflow example above, and yet many of the bugs are still schedule-dependent. No crazy Jepsen test suite needed. How can this be? All these projects have tests that validate the code on every change. Why does shuffling the schedule find so many bugs that all those tests never did?

The bugs aren’t rare; the schedules your tests run are. An idle, fast machine runs the same handful of orderings nearly every time. A test passes because the order never changes, and the code may only ever have handled that one order.

Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM?.

It takes different schedules or unforeseen long pauses, similar to the workflow story above, to trigger these bugs. The bugs are not rare, but the schedules that trigger them are. How rare are they?

The smallest race I know is counter++ from two threads.

long counter;

void increment(void) {
  counter++;
}

increment:
    mov  rax, [counter]  ; load
    add  rax, 1          ; add
    mov  [counter], rax  ; store
    ret

Two threads run increment once each. Each thread’s three steps execute in order, but the two threads can interleave. An ordering is a choice of which 3 of the 6 slots belong to thread A, so there are $\left(\genfrac{}{}{0ex}{}{6}{3}\right)=20$ of them.

We can visualize the 20 choices as paths on a grid. A step right is a step of thread A, a step down is a step of thread B, and every path from the top-left corner to the bottom-right one is one ordering.

The shaded square holds the four points where both threads have loaded the counter and neither has stored it. Any path that enters it ends with the counter at 1, because the second writer stores the stale original value.

Only the two paths along the edges provide the “correct result” of 2, A entirely before B or B entirely before A. That is 18 of 20 orderings that lose an update. If every ordering were equally likely, this race would fire in 9 runs out of 10. 😱

What does this look like on a real machine?

Let’s build a small program that runs the two threads and records the order of their steps. The program is a bit more complicated than the simple counter++ above, because it has to record the order of the steps. It also has to be careful not to introduce any new races in the recording itself. The program leverages Rewind’s rewind_point() to give Rewind VM a place to switch threads.²

/*
 * Two threads each add one to a shared counter in three steps: load the
 * counter, add one, store it back. No lock, so a thread that runs between
 * the other's load and store writes a stale value over it.
 *
 * Each step first takes the next slot of a shared atomic sequence and
 * writes its thread's letter there, so the program prints the order the
 * six steps ran in, such as AABABB, and the final counter.
 *
 * rewind_point() between the steps gives Rewind VM a place to switch
 * threads. main() calls it once before the threads start, so the one
 * failed open it makes outside Rewind happens there; on the host the
 * points inside the threads are a load and a branch.
 *
 * Built with -DSTART_BARRIER, both threads wait on a barrier before the
 * first step, so they start as close together as the host allows.
 *
 * The exit status encodes the outcome so `rewind check` reports it
 * without reading output: ordering index (0..19) * 2 + (1 if an update
 * was lost).
 */
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <string.h>

#include "rewind.h"

enum { THREADS = 2, STEPS = 3, MARKS = THREADS * STEPS };

static volatile long counter;
static atomic_int next_mark;
static char order[MARKS + 1];

#ifdef START_BARRIER
static pthread_barrier_t start;
#endif

/* Record that thread `who` is taking its next step. */
static void mark(char who)
{
	order[atomic_fetch_add(&next_mark, 1)] = who;
}

static void *increment(void *arg)
{
	char who = *(const char *)arg;

#ifdef START_BARRIER
	pthread_barrier_wait(&start);
#endif
	mark(who);
	long seen = counter; /* load */
	rewind_point();
	mark(who);
	seen = seen + 1; /* add */
	rewind_point();
	mark(who);
	counter = seen; /* store */
	return NULL;
}

/* The index of `s` among the 20 orderings of AAABBB, in sorted order. */
static int ordering_index(const char *s)
{
	int index = 0;

	for (int a = 0; a < 64; a++) {
		char candidate[MARKS + 1];
		int as = 0;

		for (int i = 0; i < MARKS; i++) {
			candidate[i] = (a >> (MARKS - 1 - i)) & 1 ? 'B' : 'A';
			as += candidate[i] == 'A';
		}
		candidate[MARKS] = '\0';
		if (as != STEPS) {
			continue;
		}
		if (strcmp(candidate, s) == 0) {
			return index;
		}
		index++;
	}
	return -1;
}

int main(void)
{
	static const char names[THREADS] = { 'A', 'B' };
	pthread_t threads[THREADS];

	/* Open /dev/rewind, or find it missing, before any thread runs. */
	rewind_point();

#ifdef START_BARRIER
	pthread_barrier_init(&start, NULL, THREADS);
#endif
	for (int i = 0; i < THREADS; i++) {
		pthread_create(&threads[i], NULL, increment, (void *)&names[i]);
	}
	for (int i = 0; i < THREADS; i++) {
		pthread_join(threads[i], NULL);
	}

	int lost = counter != THREADS;
	printf("%s %ld\n", order, counter);
	return ordering_index(order) * 2 + lost;
}

I ran both builds, with and without the barrier, 10,000 times each on my laptop, a Ryzen 7 7840U with 16 idle hardware threads.

$ for i in $(seq 10000); do ./race; done | sort | uniq -c | sort -rn
   9985 AAABBB 2
     15 BBBAAA 2
$ for i in $(seq 10000); do ./race-barrier; done | sort | uniq -c | sort -rn
   9957 BBBAAA 2
     39 AAABBB 2
      2 BBAAAB 1
      1 BAAABB 1
      1 BAABAB 1

Across 20,000 runs the idle laptop ran 5 of the 20 orderings and lost an update 4 times. Interestingly, the barrier was supposed to make them collide. Instead it flipped the order such that 9,957 runs were BBBAAA.

A busy machine is a little different. With stress-ng --cpu 16 keeping every hardware thread busy, the same two binaries ran 13 and 16 of the orderings, and lost an update in 32 and 66 of 10,000 runs. That’s still fewer than 1 run in 100 losing an update.

Rewind runs the guest on one virtual CPU and only switches threads at an exit, such as a system call (see dafuq is a deterministic VM?) or a rewind_point SDK call. This gives Rewind a chance to modify the schedule.

We can then run the same binary with Rewind to see how many of the 20 orderings it can reach.

$ rewind check --epoch 1790985600 --root root --schedules 9999 \
    --all --no-narrow -- /bin/race
...
schedule 9999: exited:0              354 steps    run 9b248f72f742b07f
1492 of 9999 other schedules ended differently

Across 10,000 runs Rewind reached 15 of the 20 orderings and lost an update in 1,371 of them. With a barrier it reached all 20 orderings and lost an update in 2,997 of them.

We can observe that Rewind’s perturbations reach more orderings than the host, and that it loses an update more often, which helps us find the bug. The table below summarizes the results:

Interleavings, though, are not the only way to trigger bugs. Long pauses are orderings too, and they can be even rarer than interleavings.

A long pause sits at the far edge of the grid. Thread A stops, and everything else takes hundreds of steps before A takes its next one. A path like that runs along one edge of the grid for a long time. A short perturbation hands the other side a few steps, so it rarely walks that far.

Rewind has knobs to make long pauses more likely. The --stall-max knob raises the maximum stall time, and the --stall-min knob lowers the minimum stall time. These settings can help explore even more unlikely schedules to discover bugs that only appear under extreme conditions.

It’s kind of surprising that with all the hype around property-based testing, fuzzing, and Jepsen, a simple deterministic VM that shuffles the schedule finds so many bugs. Our hardware is so fast that we’ve become accustomed to thinking that the order of events is always the same. Adding a little randomness to the schedule can expose bugs that have been hiding in plain sight.

Bugs aren’t rare, but the schedules that trigger them are.

When building Rewind, a deterministic VM, you come face-to-face with the idea of how to represent time very quickly. You need a way to measure time inside the guest so that on subsequent replays, the guest sees the same time and makes the same decisions. Time is the ultimate variable.

You can’t simply record the wall-clock time of the host because time spent between two interception points depends on things outside the guest, like host load, CPU frequency, caches and other processes. Therefore the same guest work can take a different amount of time on the host.

In dafuq is a deterministic VM? I explained how the clock inside Rewind VM moves 5 microseconds per step. More specifically, that clock, exit time, moves only when the guest exits such as from a syscall. It treats everything the program does between two exits as instant. Therefore a program that is CPU bound such as doing nothing but math for a full second between two system calls, sees only 5µs go by. 🤦🏻‍♂️

Rewind has a second clock mode as well, counter time, that tries to account for the work. The clock advances one nanosecond for every conditional branch the guest retired.¹

Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM?

Still confused about what I mean by “free” and “branches”? Let’s look at a simple example to distinguish between the two clocks.

The counter time tries to account for the work whereas the exit time only advanced the clock on exits.²

Why branches, though?

What is time in a replayed machine?

We already understand that we cannot rely on the wall-clock time of the host. We need something to measure that is the same for every run of the same program so that we can equate it to time spent.

What is the same on every run?

The guest’s instruction stream! The CPU is a deterministic machine: the same input, the same registers, the same memory, and the same instructions produce the same next state.

We can measure time by measuring progress through the instruction stream. 💡

Turns out that CPU manufacturers even ship a hardware counter for exactly that. Is it reliable? Let’s see. 🧐

Let’s use this simple program: count down from a billion.

int main(void) {
  long i = 1000000000;
  do {
    i--;
  } while (i != 0);
  return 0;
}

_start:
    mov  rcx, 1000000000
.loop:
    dec  rcx
    jnz  .loop
    mov  eax, 60    ; exit(0)
    xor  edi, edi
    syscall

The assembly equivalent contains a mov, a billion dec+jnz (conditional branch) pairs, then three instructions to exit for a total of 2,000,000,004 instructions.

Let’s run it on the CPU twice and count the instructions and conditional branches with perf stat:

$ nasm -felf64 loop.asm && ld -o loop loop.o
$ taskset -c 5 perf stat -e instructions:u,ex_ret_cond:u ./loop
     2,000,001,025      instructions:u
     1,000,000,000      ex_ret_cond:u

$ taskset -c 5 perf stat -e instructions:u,ex_ret_cond:u ./loop
     2,000,001,054      instructions:u
     1,000,000,000      ex_ret_cond:u

The same program and input gives the same conditional branch count, but the instruction count is off by a thousand or so. Why?

Although this loop is CPU bound and only decrements a register, it is still not isolated from the effects of the outside world. Timers tick, network packets arrive, other cores send inter-processor interrupts (IPIs). The program can’t see any of that, but the instruction counter we are probing can.

I pinned the loop to a single CPU and counted how many interrupts that CPU took during each run, from /proc/interrupts.

The dashed line is one extra count per interrupt, and the instruction counter is almost perfectly on it.³ The conditional branch counter on the other hand, stays at zero no matter how many interrupts land.

Interrupts from the outside-world are not the only thing that can skew the instruction count. The program can also page fault, which is an interrupt too. The instruction counter sees the kernel’s detour to handle the fault, but the program does not.

Tip Out of the box, AMD Zen also overcounts conditional branches, a few in tens of millions, from a speculation optimization known as SpecLockMap. The fix is to set a particular bit in the PMU MSR, which Rewind does automatically as does rr#2034.

So, what time is it?

In order to rewind history, we need a measurement of time that is reliable and deterministic. Our traditional wall-clock time is unreliable due to the effects of the outside world but we do have a reliable measurement of progress through the instruction stream: the conditional branch counter.

Using the conditional branch counter as a proxy for time turns out to be a relatively decent approximation.

Why do we care about having a “realistic” representation of time?

Plenty of programs read the clock and make decisions with it, so the guest’s clock has to be at least somewhat realistic. Under exit time, computing is “free”. A test that checks some work “finishes within 1 s” always passes. If the bug you’re chasing is a timeout, exit time will never reproduce it. 🙃

Races get skewed too with an unrealistic clock. With exit time, a whole burst of work lands in the same instant, which crowds the threads together, so a race can fire far more often in the VM than it ever would on real hardware.

The closer we can emulate the real world, the more likely we are to reproduce a bug that only happens in the real world. Counter time is a better approximation of the real world than exit time, and it is deterministic, so it is the default clock in Rewind.

Some say time is an illusion. I say it’s a jnz counter.

Despite the magic, the ease of claiming to have a bug “reproduced deterministically”, and the ease of narrowing it down to a single step, the hard part is still reading the code and understanding why it failed. Rewind can tell you what went wrong, but it cannot read the code.

This is a particular area that is ripe for AI assistance. Agents can already read the code and often find bugs. Unsurprisingly, when given a wealth of reliable data about what went wrong, they can pinpoint the bug in a way that is much more efficient than reading the code alone. The combination of deterministic runs and AI agents is incredibly powerful. 🪄

Finding the bugs in Rewind is easy. rewind check replays a test, script, or binary under many different thread schedules until one fails, then narrows the failure down to the single step. That leaves you, though, with a step number and potentially a large codebase, one you are unfamiliar with. The hard part is reading the code and understanding why it failed.

Rewind can tell you what went wrong, but it cannot read the code for you. Rewind does however give you, or an agent, a lot of information about what went wrong. I pointed Rewind VM at a cornicopia of tools, and unsurprisingly, it found bugs…many bugs. The best part is that it found them in a way that is reproducible and debuggable, which has led to the agents also easily providing the fixes as well.¹

Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM?

This is a small example of a bug that Rewind found in git, and how I used Rewind to find the bug and the line of code that caused it. The bug is in git notes, which attaches a note to a commit without changing the commit. The bug is that two notes, on two different commits, at the same time, can lose one of the notes.²

The bug

git notes attaches a note to a commit without changing the commit.

We can start with an idea for a simple test³: create two notes, on two different commits, at the same time concurrently.

git notes add -m "note 1" HEAD~0 &
git notes add -m "note 2" HEAD~1 &
wait

# both commands exited 0, so both notes should be there
git notes show HEAD~0 && git notes show HEAD~1

$ rewind check --all --schedules 32 --property exit-zero .#notes2
schedule   1: exited:1             3167 steps    run e5e5a1ecd0195f8f  violates exit-zero
schedule   2: exited:1             3312 steps    run ea6889e56564e374  violates exit-zero
...
15 of 32 other schedules violated a property

step 2682 decides it: a request to switch threads there makes the run violate exit-zero

When executed by Rewind, fifteen of thirty-two schedules lose a git-note, yet every git notes add still exits successfully.⁴

Now what?

Ask the runs, not the code

A run in Rewind is a full trace of the build: every process, every file it wrote, and any file’s contents at any step, that you replay to any point in time. That is a lot of data. Rewind does make it a little simpler to comb through the data though that help narrow the culprit.

None of those three commands knows anything about git, locks or races. They work on any program.

What ended differently?

rewind diff --outputs compares the files the two runs left behind. This is like diffoscope but on steroids 💉: it can compare any two runs, and it can compare the files at any step in the run, not just the final outputs.⁵

$ rewind diff --outputs 94c59fdf987c4944 ad0de5aec5a0be2d
3 of the 30 files the runs left behind differ:
  /build/r/.git/logs/refs/notes/commits: 36 of 465 bytes differ, first at byte 351
  /build/r/.git/refs/notes/commits: 36 of 41 bytes differ, first at byte 0
  /nix/store/014fz1lj7hmk5h3rzcaw0w4576md3bg9-git-notes2: only in the right run

Our clue is that .git/refs/notes/commits is the ref that points at the latest notes commit, and it points at different commits in the two runs.

Who wrote it, and what did it read?

rewind blame takes two runs and a file:

$ rewind blame 94c59fdf987c4944 ad0de5aec5a0be2d /build/r/.git/refs/notes/commits
commits differs at byte 0. git notes (pid 56) wrote that byte at step
2,751 in run 94c59fdf and at step 2,717 in run ad0de5ae.
...
Likely cause: git notes (pid 56) read /build/r/.git/refs/notes/commits
before git notes (pid 55) changed it at step 2,721 in run 94c59fdf,
and after that change in run ad0de5ae. It read an older version of
the file in run 94c59fdf.

In this example, Rewind has given us pretty close to the smoking gun. The second git notes add read the notes before the first one had saved its own, then saved its version over the top: a read-modify-write race.

We can visualize this exact step in the Rewind GUI at the deciding step: both git notes add processes are alive (top box), and the first is halfway through saving, with its lock file written but not yet renamed into place (bottom box).

Which line wrote it?

If we have access to debug symbols, which Nixpkgs provides, rewind where shows where in git’s own code a process was at a step. Here is pid 56 when it wrote the ref:

$ rewind where 94c59fdf987c4944 2754 --pid 56 --frames 8
process 56 (git), thread 56, at step 2754
#1 rename_tempfile (tempfile.c:347)
called from #2 commit_lock_file_to (/build/source/lockfile.h:364)
called from #3 commit_lock_file (lockfile.c:358)
called from #4 commit_ref (refs/files-backend.c:1889)
called from #5 files_transaction_finish (refs/files-backend.c:3381)
called from #6 ref_transaction_commit (refs.c:2806)
called from #7 refs_update_ref (refs.c:1592)
called from #8 commit_notes (notes-utils.c:64)

We can then look at the code at a very specific line, and try to understand why it failed. In this case, notes-utils.c:64 updates the ref without checking that it still points where it did when the command loaded the notes with a lock.

Three questions, three commands, and I read one line of git. 🤓

Who reads all of this?

The git notes failure is 2,936 steps and 1,937 events in one run, and rewind check made 33 runs. Going through one run is a lot of work, and going through all thirty-three is unrealistic.

Recently, a friend told me AI has been really useful for parsing the CAN bus of cars. The CAN bus is a firehose of messages from the car from every sensor, every button, every switch, and every light. LLMs have made managing the firehose of data easier.

I see the same emergence in Rewind. The run is also a firehose of events, and similarly, LLMs can help parse the firehose of events and find the bugs. Telling an LLM to investigate a bug with Rewind has been very effective. Narrowing the bug down to a single step and source lines, is especially helpful for an LLM as it keeps the context small and focused. At that point, the LLM can read the code and understand what went wrong, and very likely suggest a fix.

LLMs have a lot of stochasticity built into them. I have found that pairing them with more deterministic technologies such as Nix and Rewind makes them more effective. I’m very optimistic about the leveraging the combination of deterministic runs and LLMs to find and fix bugs in large codebases.

LLMs induce many of these bugs but now they can also fix them. 🙃

Since I started writing about Rewind VM, the question I get most from friends is not “does it work?” but “how does it actually work?” 🤔

D’uh! It is a virtual machine that only gets interrupted when it asks to be.

That seems to not have helped much, so let’s go deeper.

We will follow a tiny C program through the machine and, at each step, look at where non-determinism would have crept in and what Rewind does about it.

Aside: If you have not seen Rewind before, start with a flaky build you only catch once.

Life is deterministic

CPUs must be deterministic. If they were not, we would have a very bad time trusting them to do anything.

Give a CPU the same registers, the same memory and the same instructions, and it computes the same result.

What does change? Why are machines not deterministic?

They are not closed systems. They interact with the outside world and that interaction is not deterministic. A timer interrupt can land at any instruction or a disk read can complete at any time.

Aside: Computers can’t actually make random numbers. A deterministic machine can only fake it with pseudo-randomness: run the same algorithm from the same seed and you get the same numbers back. The real stuff has to come from the physical world. My favorite example is how Cloudflare keeps a wall of 100 lava lamps in its San Francisco office with a camera pointed at it, feeding its encryption keys.

If we could capture all external influences, we could replay the external world and get the same run again. That is what Rewind does.

Every source from the outside world is either removed or replaced with a value derived from a stored seed. Time, for instance, is a clock managed by Rewind. Interrupts arrive only at points we choose. ⏱️

One more outside influence is easy to miss: other CPUs. On two cores, which thread wins a race comes down to cache misses and pipeline stalls, so Rewind provides a single virtual CPU and the threads take turns on it.

A step is a VM exit

To control the outside world, Rewind needs moments where the software stops and Rewind can step in. At those moments it writes down what just happened, hands over the current time, or delivers an interrupt. So how do you stop the software at the same place on every run?

One answer is to make the hardware do it. CPUs have performance counters that can interrupt after a set number of instructions. Unfortunately, the interrupt may arrive a few instructions late, and the amount varies.¹

Rewind uses a cooperative approach: let the software tell us when something happens!

Rewind runs the guest under KVM, the hypervisor built into Linux. The guest’s code runs directly on the real CPU until it touches a device. The CPU then stops the guest and hands control to the program running the VM, Rewind’s monitor (rewind-vmm). That hand-off is a VM exit.

Rewind patches the guest kernel so that whenever it needs something only the outside world has, such as the time, a timer, or “I have nothing to do”, it makes an exit and asks the monitor. Reporting what happened, like a process starting or a line of output, is an exit too.

Each exit the guest makes is called a step, and they are numbered starting from boot.

If we assume the guest is deterministic up to step N, step N happens at the same instruction on every run.

The monitor’s loop, sketched in Python:

step, clock, deadline = 0, 0, None

while True:
    # The guest runs natively until it exits.
    exit = vcpu.run()

    # The host's own business: not a step, and nothing is sent.
    if exit.reason == HOST_INTERRUPT:
        continue

    # An exit the guest made. Time moves only here.
    step += 1
    clock += QUANTUM

    # Read the clock, arm a timer, log an event...
    deadline = handle(exit, step)

    # Interrupts go in only here, with the guest stopped.
    if deadline is not None and clock >= deadline:
        vcpu.inject(TIMER_INTERRUPT)
        deadline = None

This is true on the first run too. Nothing gets recorded, and a replay just runs the same inputs again.

Three pieces make this work, and your program doesn’t need to know about any of them.

A patched Linux kernel. The guest kernel is the piece that cooperates. A small patch teaches it to take its clock, its timer and its idle loop from the monitor instead of from hardware, and to report what processes do: exec, exit, output, files written. The monitor also hides the CPU features that would leak the outside world in, such as the hardware cycle counter and the random number instruction, so the kernel never uses them.

rewind-init. The special PID 1 is a small program that mounts the input image (a Nix closure or a root filesystem), runs the command, reports its exit status and powers the machine off. The image is mapped straight into guest memory, so reading a file is reading memory, and there is no disk to finish a read at a random moment.

rewind-vmm. The monitor owns time, decides when interrupts go in, and keeps the trace of everything the kernel reports.

The life of a system call

Here is the program. It reads the clock, draws a random number, prints both with the address of a stack variable, sleeps for a second, then reads the clock again.

#include <stdio.h>
#include <sys/random.h>
#include <time.h>
#include <unistd.h>

int main(void) {
  struct timespec ts;
  unsigned int r;

  clock_gettime(CLOCK_REALTIME, &ts);
  getrandom(&r, sizeof r, 0);
  printf("time=%ld.%09ld random=%u stack=%p\n", ts.tv_sec, ts.tv_nsec, r, (void *)&r);
  fflush(stdout);

  sleep(1);

  clock_gettime(CLOCK_REALTIME, &ts);
  printf("time=%ld.%09ld\n", ts.tv_sec, ts.tv_nsec);
  return 0;
}

On my laptop every run prints something new. In Rewind, every run prints the same thing, down to the nanosecond, until you change the seed:

$ rewind run --root root -- /bin/hello
time=1792886400.001006990 random=166963147 stack=0x7fff2bad5e4c
time=1792886401.001132684
rewind: run 85a03383f7d917d9 exited:0 after 432 steps, 1.002s virtual, 1.139s wall (poweroff)

$ rewind run --root root -- /bin/hello
time=1792886400.001006990 random=166963147 stack=0x7fff2bad5e4c
time=1792886401.001132684
rewind: run 85a03383f7d917d9 exited:0 after 432 steps, 1.002s virtual, 0.394s wall (poweroff)

# a different seed is a different input, so a different run
$ rewind run --root root --seed 1 -- /bin/hello
time=1792886400.001006990 random=2117056732 stack=0x7ffdd58e5bdc
time=1792886401.001132680
rewind: run 59c7da84895c210a exited:0 after 432 steps, 1.002s virtual, 0.848s wall (poweroff)

Same inputs, same run id, same 432 steps. Oh, and the program slept for a second, yet the whole VM finished in 0.39 seconds. 🤓

Normally, reading the time never enters the kernel: a bit of code in every process reads a hardware counter directly. Rewind hides that counter, so the read goes to the kernel, and the kernel asks the monitor.

Rewind hides the CPU’s random number instruction, so the only thing available to the guest kernel is the seed Rewind passes at boot. The kernel’s random number generator is seeded with that value, so every run of the same program with the same seed gets the same random numbers.

Most system calls never leave the guest. The few that need the outside world make an exit, and the monitor writes down what happened. The guest kernel also reports when it is idle, so the monitor can jump its clock to the next timer interrupt.²

Control those few doors and the guest is deterministic: the outside world shows up at the same step on every run.

So, dafuq is a deterministic VM? Like I said at the start, it’s a VM that only gets interrupted when it asks to be. Hopefully that makes a bit more sense now. 😅

A build is bit reproducible when the same inputs give the same bytes.

Why does reproducibility matter? They provide certainty that software is genuine and has not been tampered with. See reproducible-builds.org.

Nix is infamously not bit reproducible, but rather input-addressed: the hash of a package is calculated from the inputs rather than the output.

We still prefer though to have bit reproducibility, because it allows us to verify that the package we downloaded is the same as the one built from source. It is something many packages aspire to achieve.

When the outputs differ, you want to know why. The first step is to find the bytes that differ, and the second is to find the program that wrote them.

Finding the cause of a difference has traditionally been the challenging part of process making builds reproducible…until now!

I have been building Rewind VM, a deterministic virtual machine that can record and replay builds, and it now has a new command to find the program that wrote a byte that differs between two runs: rewind blame.

Aside: I wrote about Rewind VM first in a flaky build you only catch once, and then about properties.

Let’s see how it works, first with a small example.

A small build that is not reproducible

Let’s start with a made-up library, tiles, that writes its pkg-config file with a short Python script.¹ The script collects the packages the library links against into a set:

requires = set()
for deps in PARTS.values():
    requires.update(deps)

print(f"Requires.private: {', '.join(requires)}")

Python iterates a set of strings in an order drawn from its hash seed, which is random for every new process.

In Rewind the seed is an input: --seed N decides every random number the build draws. Let’s build tiles under eight seeds:

$ rewind check --epoch 1790985600 --seeds 8 --schedules 0 --all --no-narrow \
    github:fzakaria/rewindvm#tiles
rewind: packing 75 store paths for tiles-0.2.0
rewind: 8 random seeds, each run under schedule 0 and 0 other schedules: 8 runs
seed  0 schedule   0: exited:0       4035 steps  d20b5efd388d  run 8d29a01cc44264b4
seed  1 schedule   0: exited:0       4035 steps  b4501793e383  run c4bd9260b7719942
seed  2 schedule   0: exited:0       4035 steps  0cd5db6c5c2c  run a5448b8d2ca19c18
seed  3 schedule   0: exited:0       4035 steps  24f18f09c329  run b03811e63cc68d97
seed  4 schedule   0: exited:0       4035 steps  b4501793e383  run ed28383bea49e84f
seed  5 schedule   0: exited:0       4035 steps  5e1f0a457cbe  run f5e8d2ac2c4b7782
seed  6 schedule   0: exited:0       4035 steps  d20b5efd388d  run af24df10ae7ac1e8
seed  7 schedule   0: exited:0       4035 steps  b4501793e383  run b5268e20063c5a9e
6 of 7 other runs ended differently

The third column from the end is the start of the output’s hash (i.e. d20b5efd388d), and the last is the run id.

We did eight builds and got five different outputs. 😬 Our build is not reproducible.

rewind diff shows what differs between two runs, without leaving Rewind:

$ rewind diff --file /build/tiles/tiles.pc 8d29a01c c4bd9260
tiles.pc differs, first at line 7.
...
  Name: tiles
  Description: Tile a large image into small ones
  Version: 0.2.0
- Requires.private: pixman-1, libpng, zlib
+ Requires.private: zlib, pixman-1, libpng
  Libs: -L${libdir} -ltiles

You could ascertain this from tools like diffoscope but would be left to figure out what program wrote the differing bytes. rewind blame does that for you.

rewind blame

rewind blame takes two runs and a file, and names the program that wrote the first byte that differs:

$ rewind blame 8d29a01c c4bd9260 \
    /nix/store/ihhhzc5q...-tiles-0.2.0/lib/pkgconfig/tiles.pc
tiles.pc differs at byte 178. python3 (pid 112) wrote that byte at step
2,909 in run 8d29a01c and at step 2,909 in run c4bd9260.

22 of 227 bytes differ. The bytes from byte 178:
  8d29a01c  70 69 78 6d 61 6e 2d 31 2c 20 6c 69 62 70 6e 67  |pixman-1, libpng|
  c4bd9260  7a 6c 69 62 2c 20 70 69 78 6d 61 6e 2d 31 2c 20  |zlib, pixman-1, |

Who wrote the file in run 8d29a01c, from the last writer back:
  install (pid 122) copied it from /build/tiles/tiles.pc, starting at
    step 3,157. Its source already differs, so the copy did not cause
    the difference.
  bash (pid 112) wrote /build/tiles/tiles.pc, starting at step 2,796. It
    is the first writer, and the file already differs when it is done.
Run c4bd9260 made the file the same way, with the same programs.

In both runs, one write of 227 bytes, from byte 0, set byte 178.

python3 ran with the same command line in both runs, so the difference
comes from something it read or from randomness.

What python3 read:
  1 file that the build made, all the same in both runs:
    /build/tiles/gen-pc.py.
  31 other files that the build did not make, such as programs and
    libraries.

Likely cause: something inside python3 (pid 112), such as randomness or
the order its threads ran in. Its command line and the files the build
made that it read are the same in both runs.

The output is a little long, but it is all relevant and the first line has the answer: python3 wrote that byte at step 2,909.²

The rest of the text is how the tool determined the provenance of the byte change.

1. The file in $out was copied there by install. The file it copied from already differed, so the copy is not to blame.
2. That file, /build/tiles/tiles.pc, was first written by pid 112. It was still bash when it opened the file for the > redirect, and then became python3. One write of 227 bytes set byte 178.
3. python3 had the same command line in both runs, and the only file from the build it read, gen-pc.py, was the same in both runs.

The tool highlights that the difference was made inside python3, and we know that the seed is the one input that changed. 🎯

The fix is straightforward, we wrap the list with sorted(), and every seed then builds the same tiles.pc. 🙌

-print(f"Requires.private: {', '.join(requires)}")
+print(f"Requires.private: {', '.join(sorted(requires))}")

How it knows

How can the command deduce the lineage of a byte?

The trick is that blame does not have to search for the write. The run already recorded it.😯

Every Rewind run records each program’s starts, exits, opens for writing and renames, each at its step.³ For blame, the guest’s Linux kernel also keeps two kinds of record:

• a write record for every write to a file: which thread, which file, where in the file and how many bytes;
• an open record for every file a program opens, for reading or for writing.

The records are written to a buffer in the guest kernel’s memory, a 1MiB buffer, and Rewind copies them into trace.bin at the intervals it has stop the VM anyways. This keeps the runs identical and performant.

I measured the cost of the records in Rewind 0.9.0, and it is small enough to leave them on for every run.

With the records, blame finds the write that set the byte by reading the trace, without replaying either run. It only replays to read a file’s contents at a step.

In the app

The Rewind app shows the same thing visually. Open the two runs side by side:

$ rewind open 8d29a01c --compare c4bd9260

The Compare tab now lists the files that differ between the two runs:

We can also click a file to see it from both runs. Here that is step 2,909, with python3 gen-pc.py in the process tree and its command in the build log:

At that moment, we can explore the complete VM state: the files as they were, the process tree, a fork, or gdb. The app shows the same information as rewind blame, but visually and interactively.

Beyond seeds

In order to capture possibilities of non-determinism, we need to vary any input that can change the output. The seed decides random numbers. rewind check also varies the schedule, the order threads take turns, which is how parallel make and ninja end up with different outputs.

Turns out there is a third input that can change the output: the order of directories. The order of files in a directory is not guaranteed to be the same across machines, or even across runs on the same machine although many builds assume it to be.

We can also vary the order of directories in Rewind which helps catch even more non-determinism with --dirents. This is useful for builds that depend on the order of files in a directory.

The plugins example writes an index of the plugins it ships, in the order os.listdir returns. The example builds the same output under every seed, but not under every directory order:⁴

$ rewind check --epoch 1790985600 --seeds 1 --schedules 0 --all --no-narrow \
    --dirents 3 github:fzakaria/rewindvm#plugins
dirents  0 schedule   0: exited:0       3885 steps  77b96f07540a  run 8d64b8b46c5c8b68
dirents  1 schedule   0: exited:0       3885 steps  3d555b541296  run e4c221b26df20a9c
dirents  2 schedule   0: exited:0       3885 steps  925dde4ef23a  run 593a9430de55e469
dirents  3 schedule   0: exited:0       3885 steps  abd7877043a8  run 067e05c27c9ceb09
3 of 3 other runs ended differently

We have four directory orders, and four different outputs. Each one lists the same five plugins in another order:

The fix is the same as before: sorted(os.listdir(directory)), and every key builds the same index.

Look closely and both examples are the same bug. Python iterates a set in hash order, and ext4 lists a directory in hash order. Change the seed behind the hash and the order changes with it.

Try it

Finding the cause of a difference has traditionally been the challenging part of process making builds reproducible. Many tools exist to help detect nondeterminism but few actually give you the root cause.

Lowering the barrier to making builds reproducible is a goal of Rewind VM, and rewind blame is a step in that direction. Rewind VM is purpose built to work in the Nix ecosystem and helping make more Nix packages reproducible is a worthy goal.

$ curl -fsSL https://rewindvm.dev/install | sh
$ rewind check --seeds 8 --schedules 0 --all --no-narrow \
    github:fzakaria/rewindvm#tiles

If you want a deeper dive on this subject, check out the reproducible builds tutorial, which walks through both examples step by step, including the directory-order one.

If you maintain a package that does not reproduce and you do not know why, try blame on it, and tell me what it finds. I would love to add it to the leaderboard. 🏆

This is in the nixpkgs manual, in the Ruby section, under “Platform-specific gems” [ref]. The pull request (PR#68) it points to was opened in October 2019 and closed, unmerged, in August 2026, with: “Closing as stale. I no longer work with bundix. Feel free to take it over.” [cite]

Aside There are alternatives that have sprung up but they all have unique gaps or are not widely communicated as blessed solutions. The manual itself also does recommend a workaround for the issue, but I found it to not work in practice.

Every ecosystem has gaps everyone has learned to step around. They survive because there is just too much stuff to do. We live with these warts and gaps because the cost of fixing them is either too high or there are more important things to do.

That economics is gone.

We can have and expect amazing perfectly correct software stacks, because the cost of fixing gaps is now low enough that it is worth doing. The economics of software have changed.

The multiple days of typing are now an afternoon, or less, of directing and reading to an AI agent. In the case of Nix and Ruby, I wrote the Nix side of a Bundler lockfile solution, gemlock.nix, within an hour, removing days of misery I hit while running this blog.

While AI generated code may suffer from slop, it is a growing skill how to wield AI to produce correct verifyable code. Nix is amazingly well-suited as it gives the agents a reproducible test-bed with which to verify the correctness of the code they produce.

The economics of software have changed, and we can now expect to have no quarter for gaps in our stacks.

Despite the amazing potential this unlocks, I found few to be as enthusiastic as I was. Instead of excitement, I was met with skepticism: “What’s the point beyond it being a cool technical trick?”

First off, “The pursuit of excellence does not need justification.” by Mitchell Hashimoto should be enough.

There is however a very practical reason to pursue relocatable binaries: it allows Nix to be a package set for other build systems, specifically Bazel and buck2. The two frontends can share the same store, and the machine doing the compiling does not need to have Nix installed.

The wall

The integration of Nix and buck2 or Bazel is not novel or new. Many have tried and most have failed, or have seen mixed success.

rules_nixpkgs

Brings nixpkgs toolchains to Bazel and has been incompatible with remote execution since 2022.

buck2.nix

Brings nixpkgs toolchains to buck2 as symlinks into the local store, which works on exactly one machine.

Snowydeer

Imports buck2 outputs into Lix as content-addressed store paths, scanning the NAR for hash parts to recover the references.

The wall they all face is that a nixpkgs output is pinned to /nix/store by three absolute strings: the RUNPATH the loader reads, the PT_INTERP the kernel reads, and the symlinks that point at siblings.

A remote executor has none of those paths. you either ship the whole closure inside the executor image, mount the store over the network, or chroot every action into a fake root.

They all work but have unfortunate side effects. A new compiler version means a new executor image. An NFS mount means a shared filesystem under a system whose entire design is that it does not need one. Requiring chroot for a worker means that Nix is now a leaky abstraction.

Relocatable binaries gets rid of the wall by circumventing the problem entirely. 🤯

A closure is a layout, not a location

Absolute paths are not necessary and we can make them relative. It requires work to find all the places a path is baked in, but this is the plight of a Nix enthusiast: we patch and fix the world.

RUNPATH

Already supports relative paths; the loader expands $ORIGIN to the directory of the executable.

PT_INTERP

Linux 7.3 supports a binfmt_misc handler written in BPF that replaces $ORIGIN in the interpreter to the directory of the executable.

When we make all absolute paths relative, a beautiful thing happens: the directory nix/store/<hash>-name stops being an address and becomes a shape.

We can put the shape under /, you have a Nix store. If we put it under buck-out/v2/gen/toolchains/…/closure/, you have a toolchain artifact. If we put it under a remote worker’s input root, you have an action that runs.

It is the same beauty of what made Nix hermetic but taken to the next maximum: the shape is now relocatable. The same bytes can be in many places at once, and the loader and kernel do not care which one they are in.

Two frontends

Once a closure is a shape, the two build systems stop competing for the same job. 👏

Everyone loves Nix for its replicability but are frustrated by the needless rebuilds due to the input-address (extensional) model.

On the other hand, tools like buck2 and Bazel only allow content-addressed (intensional) inputs. Coupled with a fine-grained action graph, this allows for incremental builds and remote execution, but they don’t truly model the world of packages and their dependencies, which is where Nix excels.

Each has spent years badly imitating the other. Buck2 users hand-roll toolchain downloads; Nix users wait for a whole derivation to rebuild because one file changed.

What if Nix and buck2 could natively share the same store?

Nix could provide buck2 a closure, and buck2 would treat it like any other input. On the way out, buck2 hands Nix the name of an output, and Nix fetches it like any other store path.

The “store” in the middle is not a daemon or a filesystem. It is an agreement about layout. One build, seen through that agreement:

I have a proof-of-concept implementation of this idea, and it is surprisingly small. The BUCK file is the whole user-facing surface. The gcc closure is one buck2 tree artifact provided as a toolchain.

load("@rules//:closure.bzl", "nix_closure")
load("@rules//:cc.bzl", "cc_binary", "nix_cc_toolchain", "run_remote")

# The gcc closure, pinned by hash, as one tree artifact.
nix_closure(
  name = "closure",
  roots = {
    "gcc": "/nix/store/5q6bdx…-gcc-16.2.0",
    "glibc": "/nix/store/h4wfwi…-glibc-2.44-25",
    ...},
  # every store path the roots need, 17 of them
  closure = [...],
)

nix_cc_toolchain(name = "cc", closure = ":closure")

cc_binary(name = "hello", srcs = ["hello.c"])

# Run the binary on the worker and keep what it prints.
run_remote(name = "hello-report", exe = ":hello")

Every one of those paths is a projection of the closure artifact rather than an absolute /nix/store string, so buck2 can ship the closure wherever the compiler runs. This is most easily seen in the PT_INTERP and RUNPATH strings provided by the linker:

-Wl,--dynamic-linker=$ORIGIN/../../h4wfwi…-glibc-2.44-25/lib/ld-linux-x86-64.so.2
-Wl,-rpath,$ORIGIN/../../h4wfwi…-glibc-2.44-25/lib

hello is only told where to look within the closure, it has no idea it was a /nix/store to begin with.

The sample program hello.c reports its environment to drive home the point.

// Where did the kernel find me?
readlink("/proc/self/exe", exe, sizeof(exe) - 1);

// Which libc did the loader actually map?
dl_iterate_phdr(report_libc, out);

// Does the store path gcc was told about exist on this machine at all?
access(NIX_GCC, F_OK);

The worker is a NixOS VM running NativeLink on a 7.3 kernel with the EBPF binfmt_misc handler registered at boot. Its store is an erofs image of its own closure: 441 paths but crucially gcc is not present: ls /nix/store | grep gcc comes out empty.

$ buck2 run //:hello-report --show-output
Commands: 3 (cached: 0, remote: 3, local: 0)
BUILD SUCCEEDED

hello from a relocated store path
kernel:         Linux 7.3.0-rc7
hostname:       nix-origin-worker
/proc/self/exe: /var/lib/nativelink/work/8f7d…/buck-out/v2/gen/root/…/__hello__/root/nix/store/hello/bin/hello
libc:           /var/lib/nativelink/work/8f7d…/buck-out/v2/gen/root/…/__hello__/root/nix/store/hello/bin/../../h4wfwi…-glibc-2.44-25/lib/libc.so.6
/nix/store/5q6bdx…-gcc-16.2.0 on this machine: absent

Back into the store

Getting a buck2 output into the Nix store is where the two frontends have to agree on a name.

buck2 already has one. Every artifact it produces is identified by a digest, a Merkle hash over the output tree, and the CAS is keyed by it. Nix already supports a few ways of addressing a path by its content. Add the buck2 digest to that list and the two systems name the same bytes with the same number.

The cc_binary rule already shapes its output like mkDerivation would, nix/store/<name>/bin/hello. A buck2 output’s store path is therefore /nix/store/<digest>-hello, and both sides compute <digest> without asking the other.

The rule also knows the references, because it put glibc in RUNPATH. It writes them in a small file next to the output, together with the path and the digest. That file is the narinfo. 🤯

The way back to the host store is a substitution. The CAS is configured as a substituter, the same way cache.nixos.org is:

substituters = reapi://cluster:50051?instance=main https://cache.nixos.org

On the Nix side a buck2 target is a derivation that we reference by its digest.

packages.hello = buck2.output {
  target = "//:hello";
  # the store path is /nix/store/<digest>-hello
  digest = "sha256-…";
};

Evaluation never runs buck2: nix build asks the substituter for the pinned path and the tree streams out of the CAS, references included:

$ nix build .#hello
copying path '/nix/store/<digest>-hello' from 'reapi://cluster:50051'...
$ nix path-info --json ./result | jq -r '.[].references[]'
/nix/store/h4wfwic161kxrr74jlzla5lsm28hgary-glibc-2.44-25
/nix/store/j7qx4s4mr17j1wqgvqdzj33lmrnzb387-gcc-16.2.0-lib
$ ./result/bin/hello
argv[0]:  /nix/store/<digest>-hello/bin/hello
libc:     /nix/store/<digest>-hello/bin/../../h4wfwi…-glibc-2.44-25/lib/libc.so.6

Nothing was rewritten, rebuilt or re-hashed on the way back. The shape of the closure survived because it only ever used relative paths. The bytes the worker produced are the bytes Nix serves, under the name buck2 already knew.

The two stores become one. 🫂

A cc_binary and a mkDerivation end up as the same kind of object: a tree named by its contents, with relative paths to its neighbours. Which frontend wrote it stops mattering once it is in the store.¹

Two frontends, one store.

Patch the world

We can think big for how we want to build software. Nix affords us a world where we can make software fit beautifully together. Nix is the one ecosystem where patch the world is a reasonable plan. We can change the kernel, the loader, the compiler wrapper and the package set in a single commit.

Let’s not have static paths in the store limit our possibilities. The valhalla of incrementality awaits and we can maybe achieve it without dynamic derivations.

The yellow brick road to a two-frontend store is paved and clear:

1. nixpkgs: write relative RUNPATH. #534339 is the opt-in version; the default is the goal. The hash stays in the string so no cache is invalidated and the reference scanner keeps working.
2. nixpkgs: a location-relative cc-wrapper. The wrapper and bintools-wrapper embed absolute paths in shell. Deriving them from $(dirname "$0") turns the toolchain into something we can cp.
3. nixpkgs: relocatable = true, enforced. A fixup hook that fails any output claiming the flag while still containing the literal store prefix, in ELF or in text.
4. NixOS: the handler at boot. Once 7.3 reaches nixpkgs, a module that registers the BPF handler, gated on a PT_INTERP_NIX marker so nothing changes for binaries that did not ask.
5. Nix: one hash. A content-addressing method that matches an REAPI directory digest, so a buck2 output in the CAS is a /nix/store path.
6. Build systems: closures as artifacts. The buck2 rules in this post are a few hundred lines. The Bazel equivalent is a repository_rule away, and it would be the first rules_nixpkgs that works with remote execution.

A store path that can live anywhere is a store path everyone can use. Let’s go (re)write some strings. ✍️

I have been working on a deterministic VM called Rewind VM, where “a run” (i.e. test, command etc.) is a function of its inputs including the thread schedule. The determinism comes from the tool recording every step which lets you replay it exactly. This alone is a pretty good way to find flaky tests: run the test under Rewind, then rewind check runs it under many different thread schedules, with some slight latency added, and failures often crop up. 🐛

Feedback however was that it was not enough: the tool could find a failing schedule, but it could not tell you if that schedule was actually a bug. It only knew that the run ended differently than the original schedule, which could be a crash, a timeout, or just a different exit code. That is not enough to say “this is a bug”.

Someone pointed out that I had built most of a property-based testing tool, but I only had one property: matching exit codes.

What is property-based testing?

If you have used QuickCheck or Hypothesis, you have already used the idea. You write a property, or constraint, that must hold for every input. The tool generates lots of inputs, and when one breaks the property, it shrinks it down to the smallest input that still fails.

rewind check already did the generating and the shrinking. The inputs are thread schedules, and narrowing shrinks a failing schedule down to the one step that causes the failure. The only property though was “the run ends the same way schedule 0 did”: the same exit code and the same outputs.

That’s a pretty low bar, but easy to improve. 🙃

Rewind now lets specify properties that must be true for every run, or must be true for at least one run, or must never be true in any run.

Two tellers: Java Edition ☕

Let’s keep the bank from last time: two tellers deposit into one account from two threads, without a lock.

This time the example is in Java as a JUnit 5 suite to appeal to my Enterprise brethren.¹

void deposit(int teller, long amount) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}

A teller that runs between the other’s read and store writes a stale balance over the other’s deposit. 💥

We include a “typical” test suite: one teller, two tellers concurrently, and a new account being empty. We run it and see all tests pass and head home for the day. 🍺

Is the code correct? Let’s use Rewind to dig further.

Properties that come in the box

Before even looking at the balance, Rewind knows a lot about what a run did. Every open, write, rename, close, fork and exit is in the trace, with the step it happened at. Rewind ships properties computed from the trace alone:

Let’s throw a few at the bank and only run schedule 0:

$ rewind check --schedules 0 --property no-crash --property no-orphans \
    --property 'writes-within=/build' github:fzakaria/rewindvm#bank-java
schedule   0: exited:0     148479 steps  ba3f68ae3869  run bd3b527f0a837af8  violates writes-within
  writes-within: step 2710: process 111 (javac) opened /tmp/hsperfdata_nixbld/111 for writing, outside /build
never true in any run: sometimes "deposits-concurrent"
every schedule violated a property, as schedule 0 did

Interesting! javac writes its performance counters to /tmp/hsperfdata_<user>/<pid>². It has nothing to do with our bug, but I included it because I learned something new and it could have been a bug if it did not include the process ID.

The file properties shine for bugs between processes. The properties tutorial has a program that rewrites its config in place while another process rereads it. Its own test fails only when a read lands between the truncate and the last write, but atomic-replace flags it on the very first run. 😲

Asserting from the inside

The bank’s real invariant is not in any syscall though. It lives in memory in our example: the balance variable must add up.

For that there is the Rewind SDK, a tiny library you call from your program. Each call writes one line to /dev/rewind, a device in Rewind’s VM, and the trace keeps it with the step, process and thread that wrote it.³

// Account.deposit: another teller is mid-deposit.
int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");

// BankTest.twoTellersDeposit
Rewind.always(balance == expected, "balance-adds-up",
        () -> "balance " + balance + ", expected " + expected);

always must be true every time the program reaches it. unreachable is its opposite: a run that reaches it at all fails. These are your typical assertions, except we get the whole replay if it fails.

sometimes is the one I find the most interesting and tough to think through at times. 🤓 It tests the search: its condition must be true in at least one of the runs check tries, and it never fails a run. In model-checking terms, always is a safety property (nothing bad happens in any run) and sometimes is a reachability property (some run gets there). Here it asks whether any run had both tellers inside deposit at the same time.

Look at the output above again:

never true in any run: sometimes "deposits-concurrent"

The suite passed under schedule 0, but one teller always finished its deposit before the other started. That pass told us nothing about the race. 🤷

Let check try its default 64 schedules:

$ rewind check github:fzakaria/rewindvm#bank-java
...
true in 11 of 17 runs that checked it: sometimes "deposits-concurrent"
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate same-outcome, sdk

passing: run 61643f768ec51337, schedule 2 over steps 144694..145417
failing: run 643a298ef0ec3f7b, schedule 2 over steps 144694..145418
the two are the same run until step 145417
failing run violates same-outcome: step 149407: ended exited:2, where bd3b527f0a837af8 ended exited:0
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

That took 40 seconds on my laptop⁴ and we found that two deposits went missing.

The first line is the sometimes again: of the 17 runs check tried before it found the failure, 11 had both tellers inside deposit at once.

Did I truly fix it?

The fix is the obvious one: hold a lock from the read to the store.

int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");
synchronized (lock) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}
depositing.decrementAndGet();

$ rewind check .#bank-java
...
true in 42 of 65 runs that checked it: sometimes "deposits-concurrent"
every property held under all 65 schedules

Every schedule passes, but is that enough to make us feel warm and fuzzy?

The sometimes line is what gives me the warm embrace: in 42 of the 65 runs the tellers ran concurrently, but the lock prevented the bug. ✅⁵

A condition that is only true when the bug happens can never be true once the bug is fixed. 💡

JUnit

Java developers though love writing JUnit tests. Fear not my FizzBuzzEnterprise colleagues, Rewind has you covered: the SDK has a JUnit 5 extension.

Put @ExtendWith(RewindExtension.class) on the test class and it marks each test’s start and result.

@ExtendWith(RewindExtension.class)
class BankTest { ... }

If you can’t touch the code, junit=GLOB reads the XML reports the runner wrote instead: Maven’s surefire, Gradle, pytest’s --junitxml, whatever writes the format.

$ rewind check --property 'junit=TEST-*.xml' github:fzakaria/rewindvm#bank-java
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate junit, sdk
failing run violates junit: step 149407: test BankTest.twoTellersDeposit() passed in the unperturbed run and failed here
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

Visualizing properties

The Rewind app shows the timeline of a run, with the tests and the build log. The failing test is red, and the event card names the assertion that failed.

$ rewind open 643a298e 145713 --compare 61643f76

The strip under the timeline has a span per test: green passed, red failed. Each include a helpeful tooltip with the test name and the steps it ran.

All the other Rewind features can then be used to help isolate the cause: the source panel, gdb on a fork at any step, the thread lanes and Check from here.

Easy Peasy ?

There is a SDK for many languages already included in the Rewind repository. They are tiny, and the only dependency is the Rewind VM itself. The SDK is a single file or two per language, and you can copy it into your project.⁶

• C and C++: rewind.h, REWIND_ALWAYS(cond, "id", "fmt", ...)
• Rust: rewind_sdk::always!(cond, "id", "fmt", ...)
• Go: rewind.Always(cond, "id", "fmt", ...) and rewind.Track(t) for go test
• Python: rewind_sdk.always(cond, "id"), and a pytest plugin that loads itself
• Java: Rewind.always(cond, "id") and the JUnit 5 extension
• Shell: rewind_always id "message" test "$x" -eq 3

$ curl -fsSL https://rewindvm.dev/install | sh
$ rewind check github:fzakaria/rewindvm#bank-java

The properties tutorial goes through each piece step by step, and the advanced tutorial has script properties and a file of properties to keep beside your tests for the more adventurous.

If you happen to find any interesting bugs with Rewind let me know. I am keeping track of a small leaderboard of the best ones. 🏆