sqlelf and 20 years of Nix
If you want to skip ahead, please checkout the sqlelf repository and give me your feedback.
🎉 We are celebrating 20 years of Nix 🎉
Within that 20 years Nix has ushered in a new paradigm of how to build software reliably that is becoming more ubiquitous in the software industry. It has inspired imitators such as Spack & Guix.
Given the concepts introduced by Nix and it’s willingnes to eschew some fundamental Linux concepts such as the Filesystem Hierarchy Standard.
I can’t help but think has Nix gone far enough within the 20 years?
Making RUNPATH redundant for Nix
This post is a direct translation of Harmen Stoppel’s blog on the same subject for Nix. He has also contributed a fix to Spack.
Please check out https://github.com/fzakaria/nix-harden-needed for my solution for the Nix ecosystem.
Nix and other store-like systems (i.e. Guix or Spack), resolve all their dependencies within their store (/nix/store) to enforce hermiticity. They leverage for the most part RUNPATH which is a field on the ELF executable to instruct the dynamic linker where to discover the libraries – as opposed to searching default search paths like /lib.
❯ which ruby /nix/store/8k4sgk3bmxnj0jvcgc4wvyd8ilg0ww3y-ruby-2.7.6/bin/ruby ❯ patchelf --print-rpath $(which ruby) /nix/store/8k4sgk3bmxnj0jvcgc4wvyd8ilg0ww3y-ruby-2.7.6/lib:/nix/store/r90cncsaa519pwqpijg7ii4rkcmwjn6h-zlib-1.2.12/lib:/nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib
I have a paper about to published for SuperComputing 2022 (please reach out if you’d like early copy) that demonstrates there is a non-trivial cost to continuously searching needlessly through the RUNPATH. In fact, I have written previously about the specific costs and our tool Shrinkwrap that can avoid it.
Although Shrinkwrap is one approach for a solution, it is merely a bandaid over the existing problem.
Can systems like Nix do more to solve this problem?
Shrinkwrap: Taming dynamic shared objects
This is a blog post of a paper I have submitted for a UCSC course project.
If you are interested in the code check out https://github.com/fzakaria/shrinkwrap
One of the fundamental data management units within a Linux system are the shared object files that are loaded into memory by dynamically linked processes at startup. The mechanism and approach to which dynamic linking is done has not changed since it’s inception however software has become increasingly complex.
Computing all output paths for every attribute in Nixpkgs
Nix is an amazing tool, unfortunately doing simple things can be quite challenging.
This is a little write-up of my attempt to try and accomplish what I would have thought to be a simple thing; computing all store paths for every attribute in nixpkgs.
Why would I want to do such a thing?
I had some /nix/store entries on my system and I wanted to revisit the exact nixpkgs commit with which it was built to debug something. Without this reverse index you are pretty much out of luck for figuring it out.
I want to give early shoutout to other similar tools in this space that let you do meta searches over nixpkgs such as Nix Package Versions and Pkgs on Nix.
Using an overlay filesystem to improve Nix CI builds
Using Nix in our CI system has been a huge boon. Through Nix we have a level of guarantee of reproducibility between our local development environment and our CI platform. 🙌
Our CI infrastructure leverages containers (don’t they all now?) for each job and we explored different solutions to reduce the cost of constantly downloading the /nix/store necessary for the build.
The yak shave for reproducibility
I have been on a mission to bring reproducibility through the use of Nix into my workplace as we envision the next version of our development environment.
Similar to the movies I watch that take place in space, it only takes a small hole to destroy your hermetic environment. 🧑🚀
A Nix Binary Cache Specification
I wanted to better understand how to work with Nix binary caches. Interestingly the Nix manual which usually is a great source of knowledge, has a very limited section on the Binary Cache.
It basically just points you to nix-serve, a Perl CGI script in Eelco’s (original author of Nix) personal repository.
This guide will serve to be a loose Nix Binary Cache specification. If you are interested in browsing the end result, please checkout my OpenAPI Nix HTTP Binary Cache Specification 🎆
You can also visit the GitHub repository https://github.com/fzakaria/nix-http-binary-cache-api-spec to contribute.
Investigating hydration times with Nix
📣 I want a give a huge shoutout to my colleague Micah Catlin, whose been constantly challenging me on questions on Nix – his analytical skills are 11/10. 🤓
We are working on a new build system using Nix as the underpinning framework. In order to smooth out the developer experience, we wanted to tackle a very simple question:
“Can we force developers only to pull from the Nix binary cache?”
The search for a minimal nix-shell continued; mkShellMinimal
“The simplest things are often the truest.” - Richard Bach, 1936.
Earlier I wrote about trying to get a minimal nix-shell. The goal and challenge of the post was about reducing the dependency closure size of the shell.
I was asked what’s the point in trying to minimize the closure size ?
We are using nix-shell in our CI infrastructure and every CI job hydrates it’s own /nix/store from scratch. Reducing the dependency closure size would mean faster CI runs. 🏎️
The post finished with a question, “Can we do better ?”, to which I answered “No, not at this time.”.
I’d like to introduce mkShellMinimal that does better 🎊
A minimal nix-shell
We are currently using nix-shell to create a reproducible environment for our developers and CI infrastructure. Can we minimize the dependency closure to make our CI jobs faster?