If you just want a very easy-to-use binary cache, consider using cachix.

Nix is an amazing tool, however the learning curve can be very high. The online wiki has a lot of great documentation however I find it is often very geared towards NixOS specifically.

I wanted to better understand how to setup my own binary cache.

A binary cache builds Nix packages and caches the result for other machines. Any machine with Nix installed can be a binary cache for another one, no matter the operating system.

As mentioned above, there are a few solutions already offered in Nix:

• cache.nixos.org: The default binary cache included with all Nix installations
• cachix: A sass product that has a great free tier as long as your caches are public.
• nix-serve: A perl standalone HTTP binary cache implementation.
• Any machine can be used as a cache through SSH protocol

I specifically wanted to document & explore Nix support for binary caching using AWS S3.

The following guide assumes you have an AWS account & have setup CLI credentials .

Our custom derivation

In order to test our new cache; we’ll create a derivation that is definitely not on nixpkgs; especially the default cache service.

Let’s create a slightly modified version of the GNU hello program.

Let’s save the below derivation in a file lolhello.nix.

{ pkgs ? import <nixpkgs> { }, stdenv ? pkgs.stdenv, fetchurl ? pkgs.fetchurl }:
stdenv.mkDerivation {
  name = "lolhello";

  src = fetchurl {
    url = "mirror://gnu/hello/hello-2.3.tar.bz2";
    sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1";
  };

  patchPhase = ''
    sed -i 's/Hello, world!/hello, Nix!/g' src/hello.c
  '';
}

This guide isn’t meant to cover on how to write derivations, however hopefully this one is simple enough to follow along.

Since Nix is reproducible, the /nix/store path for the output of this derivation will always be /nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello.

Create a bucket

Le’s create a bucket at the moment to act as the root of our /nix/store.

I chose s3://fmzakari-nixcache

Generate Binary Cache Key

nix-store --generate-binary-cache-key fmzakari-nixcache \
 cache-priv-key.pem cache-pub-key.pem

We will be utilizing Nix’s ability to validate that the contents of cached paths in the store through a cryptographic signature.

Build & Sign

# build it locally so it's present in /nix/store
nix-build --no-out-link lolhello.nix
# sign the /nix/store path
nix sign-paths --key-file cache-priv-key.pem \
    $(nix-build --no-out-link lolhello.nix)

$(nix-build --no-out-link lolhello.nix) is just a quick way to return the nix/store/ output path /nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello.

Upload

# upload the contents to your S3
nix copy --to s3://fmzakari-nixcache $(nix-build --no-out-link lolhello.nix)

Purge the local store

# This deletes the /nix/store paths & the database entries
nix-store --delete $(nix-build --no-out-link lolhello.nix)

Build (from the cache!)

nix-build --no-out-link lolhello.nix
these paths will be fetched (0.03 MiB download, 0.18 MiB unpacked):
>   /nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello
> copying path '/nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello' from 's3://fmzakari-nixcache'...
> /nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello

# run the modified hello after it was pulled from the cache
/nix/store/95hmzgcfq0499l4ln72p3b4wv4smp9qw-lolhello
> hello, Nix!

If you wanted to avoid having to add the --option for nix-store or even have the caching work with nix-build, the ~/.config/nix/nix.conf file will have to updated.

Here are the contents for the same s3 cache used above however placed within the nix.conf.

substituters = https://cache.nixos.org https://looker.cachix.org s3://fmzakari-nixcache

trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= looker.cachix.org-1:9iVdEFDyfK4uFDz54S51bBuTPCSKly1PmY/tScSbja0=

There doesn’t seem to be a simple programmatic way to update nix.conf; so you’ll have to hand edit or sed :)

Using S3 was surprising a pretty straightforward way to achieve a personal binary cache; although distributing the public keys are a bit of a hassle.

Biggest pain points though seem to be:

1. Not a simple way to programmatically update the nix.conf file for the new binary caches.
2. Somewhat scary if you’d like to have multiple contributors to your new binary cache by sharing the single private key.

---