Using an overlay filesystem to improve Nix CI builds
Using Nix in our CI system has been a huge boon. Through Nix we have a level of guarantee of reproducibility between our local development environment and our CI platform. 🙌
Our CI infrastructure leverages containers (don’t they all now?) for each job and we explored different solutions to reduce the cost of constantly downloading the /nix/store necessary for the build.
The yak shave for reproducibility
I have been on a mission to bring reproducibility through the use of Nix into my workplace as we envision the next version of our development environment.
Similar to the movies I watch that take place in space, it only takes a small hole to destroy your hermetic environment. 🧑🚀
A Nix Binary Cache Specification
I wanted to better understand how to work with Nix binary caches. Interestingly the Nix manual which usually is a great source of knowledge, has a very limited section on the Binary Cache.
It basically just points you to nix-serve, a Perl CGI script in Eelco’s (original author of Nix) personal repository.
This guide will serve to be a loose Nix Binary Cache specification. If you are interested in browsing the end result, please checkout my OpenAPI Nix HTTP Binary Cache Specification 🎆
You can also visit the GitHub repository https://github.com/fzakaria/nix-http-binary-cache-api-spec to contribute.
Investigating hydration times with Nix
📣 I want a give a huge shoutout to my colleague Micah Catlin, whose been constantly challenging me on questions on Nix – his analytical skills are 11/10. 🤓
We are working on a new build system using Nix as the underpinning framework. In order to smooth out the developer experience, we wanted to tackle a very simple question:
“Can we force developers only to pull from the Nix binary cache?”
The search for a minimal nix-shell continued; mkShellMinimal
“The simplest things are often the truest.” - Richard Bach, 1936.
Earlier I wrote about trying to get a minimal nix-shell. The goal and challenge of the post was about reducing the dependency closure size of the shell.
I was asked what’s the point in trying to minimize the closure size ?
We are using nix-shell in our CI infrastructure and every CI job hydrates it’s own /nix/store from scratch. Reducing the dependency closure size would mean faster CI runs. 🏎️
The post finished with a question, “Can we do better ?”, to which I answered “No, not at this time.”.
I’d like to introduce mkShellMinimal that does better 🎊
A minimal nix-shell
We are currently using nix-shell to create a reproducible environment for our developers and CI infrastructure. Can we minimize the dependency closure to make our CI jobs faster?
Eval the Nix
I’ll keep this post short and sweet. I want to write a little blurb that I’ve been mostly enjoying using nix eval to introspect.
Java, Nix & Reproducibility
I’ve written a lot about NixOS; specifically about some of the work I’ve done to improve the Java ecosystem.
In fact, I’ve upstreamed my notes into the Java Maven Nixpkgs documentation.
🎉 Recently, the minimal ISO for NixOS has become reproducible (r13y). 🎉
The ability to produce binary reproducible artifacts is a powerful primitive. What does it take for the JVM ecosystem to adopt reproducible builds within the Nix environment?
For rationale on the why, please see https://reproducible-builds.org/
setting up a Nix Google Cloud Storage (GCS) binary cache
A previous post documented how to setup a binary cache directly on S3. Many however are tied to a different public IaaS offering and may not be able to leverage the native S3 integration Nix offers. Luckily for those using GCP, Google’s blob storage equivalent Google Cloud Storage (GCS) has interoperability with the S3 API.
This will allow us to host our binary cache on GCS while still using the native S3 integration in Nix. Following this guide will allow Nix to leverage GCS without having to use a proxy such as nix-store-gcs-proxy.
JRuby and Sorbet
A recent tweet by the JRuby folks, let me know that the work I had done a while ago to get Sorbet working seemed to have gone under the radar.
Sorbet is a fast, powerful type checker designed for Ruby.
I wanted to reflect on it’s use at our current codebase, challenges still faced and where to go next.
If you want to cut ahead and start using Sorbet right away, I’ve contributed some documentation.
