Bazel Knowledge: Secret //external directory
Did you know Bazel has a secret //external package that is created that contains
all the external repositories that are you added to WORKSPACE.bazel or MODULE.bazel ? 🤓
Let’s start with a very minimal WORKSPACE that pulls in the GNU Hello codebase.
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "gnu_hello",
urls = ["https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz"],
strip_prefix = "hello-2.10",
sha256 = "31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b",
build_file = "//third_party:gnu_hello.BUILD",
)
Bazel Knowledge: Reference targets by output name
In an attempt to try and record some of the smaller knowledge brain gains on using Bazel, I’m hoping to write a few smaller article. 🤓
Did you know you can reference an output file directly by name or the target name that produced it?
load("@bazel_skylib//rules:diff_test.bzl", "diff_test")
genrule(
name = "src_file",
outs = ["file.txt"],
cmd = "echo 'Hello, Bazel!' > $@",
)
diff_test(
name = "test_equality",
file1 = ":src_file",
file2 = ":file.txt",
)
⚠️ If the output is the same name as the rule Bazel will give you a warning but everything still seems to work.
I tend to prefer matching by rule name. I’m not yet aware of any reason to prefer one over the other.
Bazel Overlay Pattern
Do you have an internal fork of a codebase you’ve added Bazel BUILD files to?
Do you want to open-source the BUILD files (+ additional files) but doing so into the upstream project might be a bit too onerous to start? 🤔
Continuing with my dive 🤿 into Bazel for $DAYJOB$, I wanted to touch on a pattern I’ve only ever seen employed by Google for LLVM but I’m finding very powerful: Bazel Overlay Pattern.
Bazel WORKSPACE chunking
I have been doing quite a lot of Bazel for $DAYJOB$; and it’s definitely got it’s fair share of warts.
I have my own misgivings of it’s migration to bzlmod and it converging to a standard-issue dependency-management style tool.
We have yet to transition to MODULE.bazel and our codebase is quite large. As you’d expect, we hit quite a lot of diamond dependency issues & specifically with external repositories in our WORKSPACE file.
A surprising implementation detail I recently learned was how Bazel does dependency resolution for external repositories in WORKSPACE.
NixOS, Raspberry Pi & Me
I have written a lot about NixOS, so it’s no surprise that when I went to go dust off my old Raspberry Pi 4, I looked to rebrand it as a new NixOS machine.
Before I event went to play with my Pi, I was unhappy with my current home-networking setup and looked to give it a refresh.
I have had always a positive experience with Ubiquiti line of products. I installed two new AP (access points) and setup a beautiful home rack server that is completely unnecessary since my Internet provider is Comcast with top upload speeds of 35 Mbps 🥲.
Automatic Nix flake follows
If you have used Nix flakes, you likely encountered something like the following. 🤢
std.url = "github:divnix/std";
std.inputs.devshell.follows = "devshell";
std.inputs.nixago.follows = "nixago";
std.inputs.nixpkgs.follows = "nixpkgs";
hive.url = "github:divnix/hive";
hive.inputs.colmena.follows = "colmena";
hive.inputs.disko.follows = "disko";
hive.inputs.nixos-generators.follows = "nixos-generators";
hive.inputs.nixpkgs.follows = "nixpkgs";
Why is this follows necessary? 🤔
It’s in fact not necessary but it makes the Nix evaluation simpler and as a result faster. 🤓
Import but don't import your NixOS modules
This is a follow-up post to my prior one NixOS Option Inspection. Many thanks to @roberth who followed up on my issue and helped explain it. 🙏
If you are using NixOS, you’ve likely encountered the module system. It’s NixOS’s super-power and what makes it incredibly easy to share, reuse and configure systems based on Nix
{
imports = [ ./hello.nix ];
services.hello = {
enable = true;
greeter = "Bob";
};
}
In a prior post, I wrote about how it can be challenging to work backwards ⏪ from a NixOS option’s value to where it was defined.
Turns out, the answer in the post was relatively simple and Nixpkgs has a gesture for discovering the answer via definitionsWithLocations.
Turns out, in order to get definitionsWithLocations to play nice, you have to avoid a surprisingly common footgun 🥵.
NixOS Option Inspection
NixOS modules are great; and it’s one of the superpowers of NixOS. They’re so great, there was a working group to look into how to apply the concept to Nixpkgs itself.
For those uninitiated, there are plenty of guides online describing it’s value and purpose such as this one or on nix.dev.
My largest complaint thus far with it was that it’s hard to go backwards. ⏪
“Who and what defined a particular option?” 🕵️
Management Time
Note
This is the third (part1 & part2) installment on my series of improving ELF symbol relocation processing. Consider reading them if this post interests you. 📖
I have written previously about improvements to the dynamic linking process, specifically processing relocations for entries with symbols.
The key insight into the improvements, that have let the dynamic linking process scale to 1 million relocation entries and up to 22x speedup compared to the traditional methadology, is memoizing the relocation entries.
This memoization is possible since the linking order of the loader is deterministic and in store-based systems such as NixOS the libraries to be loaded are fixed & static.
Scaling past 1 million ELF symbol relocations
Note
This is a follow up to my previous post on speeding up elf relocations for store based systems.
I wrote earlier about some impressive speedups that can be achieved by foregoing the typical dynamic linking that can be applied to systems such as Nix where the dependencies for a given application are static.
