Nix S3 multi-user woes

Published 2025-07-02 on Farid Zakaria's Blog

For the longest time before embarking on my NixOS journey on my wonderful Framework 13 AMD laptop – I was a big advocate for running Nix atop a traditional Linux distribution like Debian.

I loved the simplicity of it all. I got to have my cake and eat it too. 🍰

The cherry on top was that I would install Nix in single user mode which was the default at first.

I would chown the /nix directory to my user so I wouldn’t even have to sudo. It was simple and fantastic.

Somehow along the way, the community has changed the default installation to the multi user mode which necessitates systemd and leverages a Nix daemon.

To be honest, I’m not clear why the change was made and it looks like others are just as confused. 🤨

Most uses of Nix are either on individual laptop or on ephemeral CI machines. Who are the majority of users on multi-user systems or mainframes that were the genesis for the default change?

This complexity came back recently when I tried to revive my old playbook of using AWS S3 as a binary cache – a topic I’ve written about before

I faced a variety of issues, and thought I’d write them here to hopefully save you or future me some time. 🤗

problem: I wanted to upload to my cache using nix copy on our CI runs but found that now the AWS credentials on my current user need to be pased to the daemon.

solution: Create a file at /root/.aws/credentials with the current AWS session.

sudo -E sh -c 'echo "[default]
    aws_access_key_id=$AWS_ACCESS_KEY_ID
    aws_secret_access_key=$AWS_SECRET_ACCESS_KEY
    aws_session_token = $AWS_SESSION_TOKEN" \
> /root/.aws/credentials'

Annoyingly some commands seem to use your local user and others via the daemon which complicates knowing who needs the credential, especially if it’s short lived via STS (AWS Security Token Service).

problem: I leveraged nixConfig in my flake.json but I wanted to avoid the prompt asking me to approve the binary cache on CI.

nixConfig = {
    extra-substituters = [
        "s3://my-super-secret-bucket"
    ];
    extra-trusted-public-keys = [
        "my-cache:7BzpDnjjH8ki2CT3f6GdOk7QAzPOl+1t3LvTLXqYcSg="
    ];
};

solution: Add --accept-flake-config to your nix commands.

Don’t forget, I also had to add myself as a trusted-user when I installed Nix.

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | \
    sh -s -- install linux --no-confirm --init systemd \
    --extra-conf "trusted-users = $(whoami)"

problem: I wanted to validate that my binary cache was working so I wrote a simple package to test and validated it would pull from the cache with --max-jobs 0.

pkgs.writeText "text.txt" "hello world!"

solution: Unfortunately, the trivial builder pkgs.writeText purposefully avoids substitution because it’s likely more expensive than rebuilding the file.

Use writeTextFile instead and make sure to enable allowSubstitutes.

pkgs.writeTextFile {
    name = "test.txt";
    text = "Hello World!";
    allowSubstitutes = true;
    preferLocalBuild = false;
};

problem: I want to build and cache all my homeConfigurations.

solution: Use symlinkJoin to create a meta derivation that links them all together.

homeConfigurations are not nested usually within a particular system (i.e. aarch64-linux) so I make sure to filter the set of the current system with the pkgs.system attached to a given home-manager configuration.

all = nixpkgs.legacyPackages.${system}.linkFarm "all-home-configs-${system}" (
    nixpkgs.lib.mapAttrs
    (_: config: config.activationPackage)
    (nixpkgs.lib.filterAttrs (_: config: config.pkgs.system == system) self.homeConfigurations)
);

---

Improve this page @ ecab34c
The content for this site is CC-BY-SA.