Nixpkgs maintains the highest rating on Repology for having the most packages & which are up to date. Unfortunately even with the current ecosystem of packages, there will always be gaps, and for beginners in NixOS a common question is:
“I’ve download a binary and would like to run it on NixOS”
Take a look at this graph https://repology.org/repositories/graphs
Can we do better & streamline running non-Nix software? 🤔
This was some of the questions posed by some Nix contributors and I wanted to capture the ideas put forward for others.
A brief tour of linking
Without going into a ton of detail about how dynamic libraries are performed on Linux; a Linux binary - ELF format - contains information pertaining to the dynamic libraries necessary for the binary.
For instance, here is a non-NixOS Ruby installation.
❯ readelf -d $(which ruby) | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libruby-2.7.so.2.7] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
It requires two dynamic libraries libruby & libc. These libraries may themselves have other dependencies, so we can use ldd to recursively find the dependency closure.
❯ ldd $(which ruby) linux-vdso.so.1 (0x00007ffed1705000) /lib/x86_64-linux-gnu/libnss_cache.so.2 (0x00007f3626cd0000) libruby-2.7.so.2.7 => /lib/x86_64-linux-gnu/libruby-2.7.so.2.7 (0x00007f3626960000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f362679b000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3626779000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f362676e000) libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f36266eb000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f36266e3000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f36266a8000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3626564000) /lib64/ld-linux-x86-64.so.2 (0x00007f3626cde000)
We can see here that ldd resolved the libraries to locations in my Filesystem Hierarchy Standard(FHS).
This is not hermetic, as the FHS is a global shared state across my machine. This is the exact problem that Nix itself wants to address.
I’m on a Debian distro at the moment.
Nix addresses this generally by patching the ELF header to fully specify where the shared libraries can be found in the /nix/store; so that they are not resolved or searched on the FHS.
❯ readelf -d $(which ruby) | grep RUNPATH 0x000000000000001d (RUNPATH) Library runpath: [/nix/store/z5lira1853d97gbmv1qbdjjpkjn7ksj8-ruby-2.6.6/lib: /nix/store/8fcxqg8dmwlkjw2vgkgz607kr5jy552w-zlib-1.2.11/lib: /nix/store/kah5n342wz4i0s9lz9ka4bgz91xa2i94-glibc-2.32/lib]
This patching however relies on the Nix stdenv derivation builder and ultimately is what makes binaries in Nix work.
Nix actually takes it a step further and patches the linker so that it does not even try to check the FHS.
Binaries downloaded from the Internet are not patched. What can be done?
A key insight into the bootstrapping of an ELF binary in Linux is the interpreter, whose presence is there to help satisfy any dynamic linkage.
Let’s take a look again at my non-Nix Ruby binary
❯ readelf -l $(which ruby) | grep interpreter [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
Nix built binaries use a patchelf utility that not only sets the RUNPATH to pin libraries but also changes the interpreter to one in the /nix/store
It is the interpreter’s goal to find the libraries listed in the ELF file either via the RUNPATH, LD_LIBRARY_PATH or the FHS well known directories.
⚠️ On NixOS
/lib64/ld-linux-x86-64.so.2 normally does not exist and as a result you will be greeted with an unfriendly “bad ELF interpreter: No such file or directory” error.
We have a binary that needs some shared libraries & the bootstrapping process calls out to the interpreter set in the ELF header.
💡 Let’s put a fake interpreter on NixOS machines!
This idea works since the path of Linux ld is well known for each distribution.
For instance, NixOS machines can place an entry at /lib64/ld-linux-x86-64.so.2 for a custom binary that can help resolve dynamic libraries at runtime to libraries within the /nix/store.
How can our custom ld locate the necessary libraries though? This is where we can get really crazy. 🤪
We can use nix-index – a files database for nixpkgs – to locate packages in Nix that provide the necessary library. 🤯
The packages can be realized on-demand onto the host and their /nix/store entry can then be included into the LD_LIBRARY_PATH environment variable set when handing off to the real ld.
If gc-roots are set for the required libraries, this determination can then be cached for a given binary.
Fancier best-effort matching on picking packages that have the highest % of required symbols could also be done.
It seems kind of crazy that just picking random packages from the nix-index would ultimately let us run the binary; except that is how traditional software in Linux normally works! 😱
At worst it is providing the same experience users typically experience on non-NixOS distributions but giving a gentler onboarding for people as they see the Nix-light 😇