๐ต๏ธ I did find a few other third-party solutions that either extract the data or themselves can be added as a step to your workflow to get similar visualizations. I wanted something simpler.
I previously wrote a post about the cost of runfiles which had become evident when we noticed our GitHub Bazel build workflow had slowed down by 50x.
After landing my fix, I wanted to visualize the run length time of the action; and objectively see if my fix had worked. Trust but verify.
I found the GitHub API does provide the run time length of each workflow. The GitHub CLI makes it easy to extract out the data as JSON.
The following command generates the 100 last runs of the buildAndTestBazel.yml
workflow for the openxla/stablehlo
repository. It then filters the data to only include the startedAt
and updatedAt
fields and saves it to a file called json_data.json
.
$ gh run list --workflow buildAndTestBazel.yml \
--repo openxla/stablehlo \
--json startedAt,updatedAt \
--status completed \
--branch main \
--limit 100 > json_data.json
This JSON file then served as the foundation for the next phase: visualization. Enter plotnine, a Python plotting library that brings the declarative and powerful grammar of graphics (the philosophy behind ggplot2 in R) to the Python ecosystem. Plotnine enables creating complex plots from data in a DataFrame with minimal fuss, making it an ideal choice for our purpose.
๐ต๏ธ For those unfamiliar, ggplot (and by extension, plotnine) is predicated on the idea of building plots layer by layer, adding components such as scales, axes, and geometries incrementally. This approach not only makes the plot construction intuitive but also highly customizable. I really like using them graphing because it means I can commit reusable, reliable and reproducible code for generating my visualizations.
Using another script, I transformed the JSON data into a Pandas DataFrame and then into time series graphs using plotnine. The final visualization was used to prove that my fix had worked and that the run length time had indeed decreased.
You can view the script at this gist - b3ca5a1d3700714497925d026bd8105b
import pandas as pd
from plotnine import ggplot, aes, geom_line, theme, labs, scale_x_datetime, element_text, geom_point
import json
from datetime import datetime
# $ gh run list --workflow buildAndTestBazel.yml --repo openxla/stablehlo --json startedAt,updatedAt --status completed --branch main --limit 100 > json_data.json
# Assuming `json_data` is your JSON data loaded into a variable
# If your JSON is in a file, you can load it with:
with open('json_data.json', 'r') as f:
json_data = json.load(f)
# Convert JSON data to DataFrame
df = pd.DataFrame(json_data)
# Convert the startedAt and updatedAt columns to datetime
df['startedAt'] = pd.to_datetime(df['startedAt'])
df['updatedAt'] = pd.to_datetime(df['updatedAt'])
# Calculate the elapsed time in seconds
df['elapsedTime'] = (df['updatedAt'] - df['startedAt']).dt.total_seconds()
# If your datetimes are in UTC, ensure the comparison Timestamp is also in UTC
now = pd.Timestamp.utcnow()
one_month_ago = now - pd.DateOffset(months=1)
# Filter data to the last month, ensuring both datetimes are timezone-aware or both are naive
df_last_month = df[df['startedAt'].dt.tz_localize(None) >= one_month_ago.tz_localize(None)]
# Plot using plotnine
plot = (ggplot(df_last_month, aes('startedAt', 'elapsedTime')) +
geom_line() +
theme(axis_text_x=element_text(rotation=90, hjust=1)) +
labs(x='Date Workflow Started', y='Elapsed Time (seconds)', title='Elapsed Time Series') +
scale_x_datetime(date_breaks='1 day', date_labels='%Y-%m-%d')) + geom_point()
# Display or save the plot
# To display the plot in a Jupyter notebook or similar environment, just use:
plot.show()
# To save the plot to a file, use:
plot.save("elapsed_time_series.png", width=4, height=4, dpi=150)
๐ฒ ChatGPT & Gemini were both pretty good options at generating the code necessary to generate the graphs. Declarative language for tasks proves to be a powerful use for LLMs.
The visualized graph may not be the prettiest but it gets the job done and is reproducible for the dataset.
If you have suggestions on how to improve the visualization or the process, please let me know. Hopefully GitHub will provide a built-in way to visualize this data in the future. ๐ค
]]>tl;dr; This is a little story about how making Bazel hermetic can lead to some unexpected consequences. In this particular case, it caused our GitHub action to slow down by 50x from 1 minute to over 60 minutes.
The fix recommended was to apply the following to your .bazelrc
โ I needed to understand why however.
# Disabling runfiles links drastically increases performance in slow disk IO situations
# Do not build runfile trees by default. If an execution strategy relies on runfile
# symlink tree, the tree is created on-demand. See: https://github.com/bazelbuild/bazel/> > issues/6627
# and https://github.com/bazelbuild/bazel/commit/03246077f948f2790a83520e7dccc2625650e6df
build --nobuild_runfile_links
test --nobuild_runfile_links
# https://bazel.build/reference/command-line-reference#flag--legacy_external_runfiles
build --nolegacy_external_runfiles
test --nolegacy_external_runfiles
Bazel is a popular build system for those seeking to adopt Google-style build methodologies & kool-aid with the hopes of achieving hermetic nirvana. Very quickly after adopting Bazel, you realize that although youโve defined your build targets, ultimately the default toolchains provided by Bazel use the system provided binaries, libraries and headers.
๐ NixOS is a great way to bring in a reproducible environment for Bazel to use.
Recently, I decided to โfixโ our non-hermetic Python Bazel builds by adding a new reliance on @python_rules which helps pull in a Python toolchain for Bazel to use.
๐ Toolchains in Bazel are often downloaded as pre-compiled binary blobs, so their reproducibility is still limited to and can vary according to the underlying system being run.
Python is a toolchain but also a runtime file โrunfileโ, as the language is interpreted. The result of including @python_rules was that every Python test target (py_test) included a runfile tree for a complete Python installation.
Here is a sample of the symlinks for the Python installation that are created.
โฏ ls -l bazel-out/k8-fastbuild/mytest.test.runfiles/rules_python\~0.30.0\~python\~python_3_10_x86_64-unknown-linux-gnu/bin
2to3 -> /home/fmzakari/.cache/bazel/_bazel_fmzakari/17bce12c4b47a4a2fc75249afee05177/external/rules_python~0.30.0~python~python_3_10_x86_64-unknown-linux-gnu/bin/2to3
2to3-3.10 -> /home/fmzakari/.cache/bazel/_bazel_fmzakari/17bce12c4b47a4a2fc75249afee05177/external/rules_python~0.30.0~python~python_3_10_x86_64-unknown-linux-gnu/bin/2to3-3.10
idle3 -> /home/fmzakari/.cache/bazel/_bazel_fmzakari/17bce12c4b47a4a2fc75249afee05177/external/rules_python~0.30.0~python~python_3_10_x86_64-unknown-linux-gnu/bin/idle3
How many files (inodes) are created for each test?
โฏ find bazel-out/k8-fastbuild/bin/stablehlo/tests/transform_chlo.mlir.test.runfiles/rules_python\~0.30.0\~python\~python_3_10_x86_64-unknown-linux-gnu | wc -l
2458
๐คฏ 2458 ๐คฏ
Let that sink in. Bazel will create 2458 inodes (symlinks) for each test target. It will do this by default for builds even if the tests are never run or executed.
To double down on this pain, Bazel supports two runfile trees for external repositories which is outlined in their wiki.
๐คฏ So the whole tree exists at least twice for each test target. ๐คฏ
This increase in symlinks caused our GitHub cache action to suddenly jump from 1 minute to over 60 minutes. This was a 50x increase in time to run the action. The tar and untar of the .cache
directory had to process so many additional files that it was an IO bottleneck.
The recommended approach (from Bazel Slack and other web links) is to have the following in your .bazelrc
# Disabling runfiles links drastically increases performance in slow disk IO situations
# Do not build runfile trees by default. If an execution strategy relies on runfile
# symlink tree, the tree is created on-demand. See: https://github.com/bazelbuild/bazel/issues/6627
# and https://github.com/bazelbuild/bazel/commit/03246077f948f2790a83520e7dccc2625650e6df
build --nobuild_runfile_links
test --nobuild_runfile_links
# https://bazel.build/reference/command-line-reference#flag--legacy_external_runfiles
build --nolegacy_external_runfiles
test --nolegacy_external_runfiles
bazel test
or bazel run
which sets up the runfile tree.Bazel can provide some amazing guarantees but navigating the myriad of knobs can be frustrating.
Having had experience with NixOS โ Iโm curious why Bazel doesnโt support template support for the shebang to point to a single location for the Python interpreter.
Anyways, if you want to make your GitHub actions faster, consider adding the above to your .bazelrc
file.
I did not discover or invent this trick .
I wanted to make available a Python wheel to some developers but I did not want to publish it on PyPI for a variety of reasons.
The trick is pretty simple but leverages two simple facts:
For a URL to behave similar to PyPI for pip
to install a package, it merely must provide an index.html
file with links to the wheels.
This is the premise of PEP 503 which defines the PyPI Simple Repository API.
<!DOCTYPE html>
<html>
<body>
<a href="/frob/">frob</a>
<a href="/spamspamspam/">spamspamspam</a>
</body>
</html>
GitHub Release Page has a view that includes all the links to all assets in the release. For instance for letโs consider the mlir-wheels repository that uses this trick. It has a single release with over 5,000 โassetsโ, where each asset is a wheel for a different version and particular platform.
pip
itself cannot use this page unfortunately, because the hyperlinks are loaded via Javascript.
There is however an alternative page that is a basic HTML view of all the assets.
https://github.com/makslevental/mlir-wheels/releases/tag/latest -> https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest
With this page you can easily use pip
to install and upgrade the packages.
pip install mlir-python-bindings \
-f https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest
Happy hosting. ๐
]]>Please checkout the sqlelf repository and give me your feedback.
I wrote in my earlier post about releasing sqlelf. I had the hunch that the existing tooling we have to interface with our object formats, such as ELF, are antiquated and ill specified.
Declarative languages, such as SQL, are such a wonderful abstraction to articulate over data layouts and let you reason about what you want rather than how to get it.
Since continuing to noodle ๐จโ๐ป on the project, Iโve been pleasantly surprised at some of the introspection, specifically with respect to symbols, Iโve been able to accomplish with SQL that would have been a pain using traditional tools.
Come on a stroll with me on a few case studies Iโve gone through on how SQL guided analysis wins out.
One of the primary data structures within the ELF file is the symbol table, especially the dynamic symbol table that allows the use of shared objects (libraries).
A typical question someone may ask themselves though is:
Which library that I load is providing function foo?
This is a worthwhile question because you would like to know which shared object is not only providing the symbol definition but also which the linker (ld.so
) choses
to link against at runtime.
The state of the art (prior to sqlelf) of how to retrieve this diagnostic information is using LD_DEBUG
environment variable and trolling through the large dump of logs it emits. ๐คฆ
โฏ LD_DEBUG=symbols,bindings /usr/bin/ruby |& head
1228310: symbol=__vdso_clock_gettime; lookup in file=linux-vdso.so.1 [0]
1228310: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
1228310: symbol=__vdso_gettimeofday; lookup in file=linux-vdso.so.1 [0]
1228310: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_gettimeofday' [LINUX_2.6]
1228310: symbol=__vdso_time; lookup in file=linux-vdso.so.1 [0]
1228310: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_time' [LINUX_2.6]
1228310: symbol=__vdso_getcpu; lookup in file=linux-vdso.so.1 [0]
1228310: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_getcpu' [LINUX_2.6]
1228310: symbol=__vdso_clock_getres; lookup in file=linux-vdso.so.1 [0]
1228310: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_getres' [LINUX_2.6]
Letโs see how we can re-think of this question as a declarative SQL statement:
SELECT caller.path as 'caller.path',
callee.path as 'calee.path',
caller.name,
caller.demangled_name
FROM ELF_SYMBOLS caller
INNER JOIN ELF_SYMBOLS callee
ON
caller.name = callee.name AND
caller.path != callee.path AND
caller.imported = TRUE AND
callee.exported = TRUE
We can think of the above ask asking:
Please provide all pairings of symbols where the name is the same between any two different ELF files. One of the files must export the symbol and the other must be importing it.
โฏ sqlelf /usr/bin/ruby --sql "SELECT caller.path as 'caller.path',
callee.path as 'calee.path',
caller.name
FROM ELF_SYMBOLS caller
INNER JOIN ELF_SYMBOLS callee
ON
caller.name = callee.name AND
caller.path != callee.path AND
caller.imported = TRUE AND
callee.exported = TRUE
LIMIT 10"
โโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโ
โ caller.path โ calee.path โ name โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libruby-3.1.so.3.1.2 โ ruby_run_node โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libruby-3.1.so.3.1.2 โ ruby_init โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libruby-3.1.so.3.1.2 โ ruby_options โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libruby-3.1.so.3.1.2 โ ruby_sysinit โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libc.so.6 โ __stack_chk_fail โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libruby-3.1.so.3.1.2 โ ruby_init_stack โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libc.so.6 โ setlocale โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libc.so.6 โ __libc_start_main โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libc.so.6 โ __libc_start_main โ
โ /usr/bin/ruby โ /usr/lib/x86_64-linux-gnu/libc.so.6 โ __cxa_finalize โ
โโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโ
๐ฅณ That sure beats dealing with unstructured text. sqlelf can also emit a myriad of output formats such as csv or json.
The reality of LD_DEBUG
however is that it usefulness came in knowing the final resolution of a given symbol.
ELF files are free to have and export any symbols and often times, wether by accident (benign) or maliciously, the same symbol may be exported by multiple libraries.
This is what empowers tools such as
LD_PRELOAD
so that users can take over symbols such asmalloc
and replace them with alternative strategies.
The linker, according to the SystemV ABI, examines the symbol tables with a breadth-first search across the dependency graph of the shared object libraries.
A typical question someone may ask themselves though is:
What symbols are currently shadowed in my dependency graph?
Letโs see how we can re-think of this question as a declarative SQL statement:
SELECT name, version, count(*) as symbol_count,
GROUP_CONCAT(path, ':') as libraries
FROM elf_symbols
WHERE exported = TRUE
GROUP BY name, version
HAVING count(*) >= 2
We can think of the above ask asking:
Please provide me all symbols (and the library that defines them) that are exported by more than 2 libraries.
Any symbol here is technically being shadowed, whether on purpose or benign.
Revisiting the same ruby example above we can see the results.
โฏ sqlelf /usr/bin/ruby --recursive --sql "
SELECT name, version, count(*) as symbol_count,
GROUP_CONCAT(path, ':') as libraries
FROM elf_symbols
WHERE exported = TRUE
GROUP BY name, version
HAVING count(*) >= 2"
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ name โ version โ symbol_count โ libraries โ
โ __finite โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ __finitef โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ __finitel โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ __signbit โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ __signbitf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ __signbitl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ copysign โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ copysignf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ copysignl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ finite โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ finitef โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ finitel โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ frexp โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ frexpf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ frexpl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ ldexp โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ ldexpf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ ldexpl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ modf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ modff โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ modfl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ scalbn โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ scalbnf โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โ scalbnl โ GLIBC_2.2.5 โ 2 โ /usr/lib/x86_64-linux-gnu/libm.so.6:/usr/lib/x86_64-linux-gnu/libc.so.6 โ
โโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
In this particular case, there is no malicious symbol shadowing and the symbols shadowed by libm
and libc
are well
known. In fact, on many systems, libm
is a symlink to libc
.
I have previously written about through the development of shrinkwrap that a more annoying shadowing can happen with OpenMPI. Itโs pretty easy to accidentally get the no-op library implementation earlier in breadth-first search and find yourself with a sequential application.
Iโve included a neat example in the sqlelf repository that you can play with to test shadowing symbols and see the results of sqlelf. ๐ต๏ธ
If you find any of this fascinating, contribute and letโs work to make accessing ELF via SQL simple and productive.
]]>If you want to skip ahead, please checkout the sqlelf repository and give me your feedback.
๐ We are celebrating 20 years of Nix ๐
Within that 20 years Nix has ushered in a new paradigm of how to build software reliably that is becoming more ubiquitous in the software industry. It has inspired imitators such as Spack & Guix.
Given the concepts introduced by Nix and itโs willingnes to eschew some fundamental Linux concepts such as the Filesystem Hierarchy Standard.
I canโt help but think has Nix gone far enough within the 20 years?
If you have kept an eye on some of the work Iโve been doing and thinking of, I have spent some time thinking how Nix can make further progress on goals of reliability and reproducibility.
If you havenโt seen my talk on Rethinking basic primitives for store based systems, I recommend you watch it.
Nix, and more specifically NixOS, is uniquely poised to do-away with many of the historic cruft that has plaque us in software due to the fact that itโs dependency closure goes down to the Linux kernel!
There is no short-list of components we can re-imagine however I have been focused on the dynamic linker / interpreter. Concepts of the *Unixes of the world which are largely historic are up for grabs.
As part of my work on Shrinkwrap, I was getting pretty frustrated working with the ELF file format.
Checkout my SuperComputing 2022 paper Mapping Out the HPC Dependency Chaos
The best tools we have to introspect binaries are readelf
and objdump
whom simply dump raw ASCII text to the console.
โฏ readelf --demangle --dyn-syms /usr/bin/ruby | head
Symbol table '.dynsym' contains 22 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND ruby_run_node
Why are we restricted to such a challenging file format to work with?
I am working on an idea which I am very excited about ๐ค which I will be writing about. To prove to myself that the idea has merit, I wanted to explore making a tool that allows for easier introspection of ELF files.
Using the power of SQLite and the Virtual Tables concept, I wrote sqlelf: Explore ELF objects through the power of SQL.
โฏ sqlelf /usr/bin/ruby /bin/ls /usr/bin/pnmarith
sqlite> SELECT elf_headers.path, COUNT(*) as num_sections
..> FROM elf_headers
..> INNER JOIN elf_sections ON elf_headers.path = elf_sections.path
..> WHERE elf_headers.type = 3
..> GROUP BY elf_headers.path;
path|num_sections
/bin/ls|31
/usr/bin/pnmarith|27
/usr/bin/ruby|28
If I can prove a clean 1:1 mapping between the two formats then there is an amazing room for potential.
I am still working through the domain model mapping to individual tables (contributions and help appreciated!) but I am really excited by this idea.
Unix introduced the simple concept of what if everything was a file?
๐ ๐ ๐ What if everything was a database? ๐ ๐ ๐
]]>This post is a direct translation of Harmen Stoppelโs blog on the same subject for Nix. He has also contributed a fix to Spack.
Please check out https://github.com/fzakaria/nix-harden-needed for my solution for the Nix ecosystem.
Nix and other store-like systems (i.e. Guix or Spack), resolve all their dependencies within their store (/nix/store) to enforce hermiticity. They leverage for the most part RUNPATH which is a field on the ELF executable to instruct the dynamic linker where to discover the libraries โ as opposed to searching default search paths like /lib.
โฏ which ruby
/nix/store/8k4sgk3bmxnj0jvcgc4wvyd8ilg0ww3y-ruby-2.7.6/bin/ruby
โฏ patchelf --print-rpath $(which ruby)
/nix/store/8k4sgk3bmxnj0jvcgc4wvyd8ilg0ww3y-ruby-2.7.6/lib:/nix/store/r90cncsaa519pwqpijg7ii4rkcmwjn6h-zlib-1.2.12/lib:/nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib
I have a paper about to published for SuperComputing 2022 (please reach out if youโd like early copy) that demonstrates there is a non-trivial cost to continuously searching needlessly through the RUNPATH. In fact, I have written previously about the specific costs and our tool Shrinkwrap that can avoid it.
Although Shrinkwrap is one approach for a solution, it is merely a bandaid over the existing problem.
Can systems like Nix do more to solve this problem?
Nix and similar systems have the benefit of knowing ahead of time the exact path for every dependency. They are also particularly well positioned to make deep impactful changes because they rebuild the world bottom up, starting from libc.
Although these systems are radical departures from traditional Linux distributions and try to rebel against the Filesystem Hierachy Standard, they nevertheless rely on fundamental tooling and control knobs such as RUNPATH to bandaid over the solution.
๐ง Letโs make the use of RUNPATH in Nix obsolete and unnecessary.
We can do this through the observation that GCC will propagate the soname stored in the library into the DT_NEEDED entry of the upstream binary. That means that if our soname happens to be an absolute path, such as a /nix/store entry, it will get set for the DT_NEEDED and searching through RUNPATH will be unnecessary. ๐
Here is a small example to demonstrate
# Let's build our library with an absolute path for soname
# notice I have set the soname to an absoulte path
โฏ gcc -shared -o libf.so -Wl,-soname,/nix/store/znxycsxlnx2s9zn6g0s0fl4z57ar7aps-libf-0.1/lib/libf.so -x c - <<EOF
#include <stdio.h>
void f() { puts("hello world"); }
EOF
โฏ patchelf --print-soname libf.so
/nix/store/zir4jfm86i3037lnsaz5br55iwavvhpz-libf-0.1/lib/libf.so
# now build the application that relies on it
โฏ gcc -o app -lf -L. -x c - <<EOF
void f();
int main() { f(); }
EOF
โฏ patchelf --print-needed app
/nix/store/znxycsxlnx2s9zn6g0s0fl4z57ar7aps-libf-0.1/lib/libf.so
libc.so.6
Nothing of the above cannot be done during the building of a library automatically within Nix and specifically Nixpkgs. During build time, the store path is known ($out) and it can be set for the soname.
Ideally, I believe a deep fix for Nixpkgs is possible by altering the ld-wrapper to correctly set the soname which will be picked up for every derivation built with stdenv.
Unfortunately since I am not a bash Guru, I opted for a fix similar to autopatchelf โ nix-harden-needed.
It is a Nix stdenv setuphook that will automatically call patchelf to set the soname to the correct value โ their absolute path. This means that the DT_NEEDED entry for any binary upstream that relies on it will leverage the absolute path and avoid the costly lookup through RUNPATH.
Itโs incredibly easy to use, you simply have to add it as a build input to your derivation. Letโs revisit the above example using the setup-hook.
let libf = stdenv.mkDerivation rec {
pname = "libf";
version = "0.1";
dontUnpack = true;
buildInputs = [
nix-harden-needed-hook
];
buildPhase = ''
# Enable if you'd like to see wrapper debug information
# NIX_DEBUG=1
$CC -shared -o libf.so -Wl,-soname,libf.so -x c - <<EOF
#include <stdio.h>
void f() { puts("hello world"); }
EOF
'';
installPhase = ''
mkdir -p $out/lib
mv libf.so $out/lib
'';
};
in
stdenv.mkDerivation rec {
pname = "app";
version = "0.1";
dontUnpack = true;
buildInputs = [
libf
];
buildPhase = ''
# Enable if you'd like to see wrapper debug information
# NIX_DEBUG=1
$CC -o app -lf -x c - <<EOF
void f();
int main() { f(); }
EOF
'';
installPhase = ''
mkdir -p $out/bin
mv app $out/bin
'';
}
The built binary will correctly have the DT_NEEDED set to the absolute path of the shared object file.
โฏ patchelf --print-needed /nix/store/6pg9d3lwlmgcmmswv937fcy211vkqxch-app-0.1/bin/app
/nix/store/znxycsxlnx2s9zn6g0s0fl4z57ar7aps-libf-0.1/lib/libf.so
libc.so.6
โฏ patchelf --print-soname /nix/store/zir4jfm86i3037lnsaz5br55iwavvhpz-libf-0.1/lib/libf.so
/nix/store/zir4jfm86i3037lnsaz5br55iwavvhpz-libf-0.1/lib/libf.so
I believe systems such as Nix have ushered a new paradigm shift of thinking about software and there is immense opportunity to go beyond the current limitations and tooling.
๐ Everything can be re-thought and re-imagined.
]]>Please reach out if you are a bash guru and we can work together to apply a deeper fix to Nixpkgs of the above ideas.
This is a blog post of a paper I have submitted for a UCSC course project.
If you are interested in the code check out https://github.com/fzakaria/shrinkwrap
One of the fundamental data management units within a Linux system are the shared object files that are loaded into memory by dynamically linked processes at startup. The mechanism and approach to which dynamic linking is done has not changed since itโs inception however software has become increasingly complex.
This is the full build and run closure for Ruby in Nix, which is a good visual depiction of the complexity.
The discovery of the needed dependencies can at most be controlled by small set of directory lists or typically rely on convention for discovery, better known as the Filesystem Hierarchy Standard (FHS).
The reliance on convention for discovery of shared objects while simple, has resulted in challenges when trying to rebuild solutions reproducibly and root cause discrepancies between machines โ โIt works on my machine but not yoursโ.
Novel new software packaging models have emerged such as Nix, Guix and Spack, that attempt to tame the chaos of dependency hell by eschewing all uses of the FHS and relying on explicit deterministic paths. These tools have made great strides in moving software packaging to becoming more reproducible but still exhibit certain flaws, specifically performance, as a result of building upon tooling that was designed for a different paradigm.
Shrinkwrap is a tool that attempts to overcome some of the performance limitations with how software may be packed in store-like models by freezing the dependencies directly on the executable.
For a different approach to this problem, check out this blog post by the Guix developers.
The number of dependencies needed for a particular binary transitively, and the RUNPATH
can vary greatly. For instance, emacs lists 36 directories in itโs RUNPATH
and requires 103 dependencies to be resolved.
The result is that the dynamic linker must attempt potentially 3600 filesystem operations (openat or stat) to resolve the needed dependencies every time the process is started.
๐ This exorbitant cost can be made worse if the store itself resides on a shared filesystem such as NFS. ๐
$ patchelf --print-rpath /nix/store/vvxcs4f8x14gyahw50ssff3sk2dij2b3-emacs-27.2/bin/.emacs-27.2-wrapped \
| tr ':' '\n' | wc -l
36
$ ldd /nix/store/vvxcs4f8x14gyahw50ssff3sk2dij2b3-emacs-27.2/bin/.emacs-27.2-wrapped | wc -l
103
๐ก When faced with a recurring problem, often the solution is to cache the previous answer to avoid unecessary work.
Shrinkwrap adopts this approach by freezing the required dependencies directly into the DT_NEEDED
section of the binary by having it point to an absolute path. The
transitive dependency list is also lifted to the top-level binary to simplify auditing the required dependencies.
$ patchelf --print-needed /nix/store/zb2h75vbhg7w42b3f42bl0y2d4m0a4n3-emacs-27.1/bin/.emacs-27.1-wrapped
libtiff.so.5
libjpeg.so.62
libpng16.so.16
libz.so.1
libungif.so.4
libXpm.so.4
libgtk-3.so.0
libgdk-3.so.0
$ shrinkwrap /nix/store/zb2h75vbhg7w42b3f42bl0y2d4m0a4n3-emacs-27.1/bin/.emacs-27.1-wrapped -o emacs_stamped
$ patchelf --print-needed emacs_stamped
/nix/store/2nkjrh3za68vrw6kf8lxn6nq1dval05v-gcc-10.3.0-lib/lib/libstdc++.so.6
/nix/store/jvbyjnjh4w8qg7izfq4x5d2wy9lv9461-icu4c-70.1/lib/libicudata.so.70
/nix/store/2kzsm8hhc4lzji6g1ksav9bdjbbiyxln-libgpg-error-1.42/lib/libgpg-error.so.0
/nix/store/mpwncqr8fbqflmglkrxj7a288xdbymk3-util-linux-2.37.2-lib/lib/libblkid.so.1
/nix/store/8n6mjngkw6909rx631rzwby2rsdk0blf-libglvnd-1.3.4/lib/libGLX.so.0
/nix/store/8n6mjngkw6909rx631rzwby2rsdk0blf-libglvnd-1.3.4/lib/libGLdispatch.so.0
/nix/store/xlvnyyviqcjys8if5hgkyykgv7d10hb8-libdatrie-2019-12-20-lib/lib/libdatrie.so.1
/nix/store/2zl3dw54ysdf55hngapkkfhiw0w8c9gp-json-glib-1.6.6/lib/libjson-glib-1.0.so.0
/nix/store/30q5xa4pfbvic54nh68qn86w6kjki66i-sqlite-3.36.0/lib/libsqlite3.so.0
/nix/store/jvbyjnjh4w8qg7izfq4x5d2wy9lv9461-icu4c-70.1/lib/libicui18n.so.70
/nix/store/jvbyjnjh4w8qg7izfq4x5d2wy9lv9461-icu4c-70.1/lib/libicuuc.so.70
Applying Shrinkwrap resulted in a large reduction in syscalls, which equates to a 36x speedup. The absolute amount recovered may seem negligible however this unecessary penalty is paid on every process invocation, and on every machine executing the binary.
Program | Calls(stat/openat) | Time (Seconds) |
---|---|---|
emacs | 1823 | 0.034121 |
emacs_stamped | 104 | 0.000950 |
The above was captured using
strace โ strace -e openat,stat -c ./emacs_stamped --version
Shrinkwrap relies on the ability for a dynamic linker to deduplicate libraries with a common basename or whose soname (ELF header value) are the same. For instance in the below image, Shrinkwrap elevated libac.so
to a direct absolute dependency of the binary, but relies on the dynamic linker deduplicating the resolution for libxyz.so
which does not refer to it absolutely.
โ ๏ธ This functionality currently does not exist in musl and only works with glibc. โ ๏ธ
Please see this mailing list discussion for more details with musl.
Nothing in Shrinkwrap assumes any Nix specifics and it may also be integrated into other store-like systems as well such as Guix and Spack.
It is not yet integrated into Nixpkgs but I woud love feedback. ๐
Changing the needed dynamic dependencies to point to absolute paths, especially when those paths are immutable and content-addressable, may have philosophical and legal considerations for certain open-source licenses such as LGPL.
LGPL specifically mentions that only in the case of dynamic linking is the license not propagated over. Although these dependencies go through the process of being dynamically linked, the library they are linked to is effectively fixed.
Does this distinction blur the differentiation between static and dynamic linking?
What if the linker validated the content-address to also verify the library hasnโt been changed?
Additional investigation into the legal ramifications may be an opportunity for future work.
]]>This is a little write-up of my attempt to try and accomplish what I would have thought to be a simple thing; computing all store paths for every attribute in nixpkgs.
Why would I want to do such a thing?
I had some /nix/store entries on my system and I wanted to revisit the exact nixpkgs commit with which it was built to debug something. Without this reverse index you are pretty much out of luck for figuring it out.
I want to give early shoutout to other similar tools in this space that let you do meta searches over nixpkgs such as Nix Package Versions and Pkgs on Nix.
๐ฃ๏ธ This goal has been made extra arduous due to the migrations of the new Nix CLI commands and the migration to Flakes.
Any attempt to try and do this within the Nix language expression is doomed. Any immediate attempt to iterate over all keys within the top-level attribute in Nixpkgs is faced with hurdle and after hurdle.
I got pretty far with the following gist but it still hit roadblocks.
let
pkgs = import <nixpkgs> {
config.allowBroken = true;
config.allowUnfree = true;
};
lib = import <nixpkgs/lib>;
tryEval = builtins.tryEval;
in
lib.mapAttrs (k: v:
let name = (tryEval v.name or "");
out = (tryEval v.outPath or "");
in {
name = name.value;
out = out.value;
}
) pkgs
Feedback from the community has been to use nix tooling such as nix-env or nix-search that have special handling for all these sharp edges, or some of the fancier work used by hydra or ofborg.
I succumb to peer pressure and decided to use these tools ๐ฎโ๐จ rather than what I was hoping to be an elegant pure Nix expression.
โฏ nix search . --json | jq -r 'keys|.[]' > package-names.txt
โฏ head -n 10 package-names.txt
legacyPackages.x86_64-linux.AMB-plugins
legacyPackages.x86_64-linux.ArchiSteamFarm
legacyPackages.x86_64-linux.AusweisApp2
legacyPackages.x86_64-linux.CHOWTapeModel
legacyPackages.x86_64-linux.ChowKick
legacyPackages.x86_64-linux.CoinMP
legacyPackages.x86_64-linux.CuboCore.coreaction
legacyPackages.x86_64-linux.CuboCore.corearchiver
legacyPackages.x86_64-linux.CuboCore.corefm
legacyPackages.x86_64-linux.CuboCore.coregarage
๐ Super annoying that these new Nix commands even with -L continue to write the text in-place for their log output.
Since we are using nix search the names of the packages are now following the Flakes naming convention with prefixing them with legacyPackages.
The plan here to continue with the new Nix commands and now evaluate the outPath of each package.
โ ๏ธ This does not build the derivation.
โฏ cat package-names.txt | \
xargs -I'{}' sh -c \
'nix eval --raw ".#{}.outPath" >> outpaths.txt; echo >> outpaths.txt'
โฏ head -n 10 outpaths.txt
/nix/store/jrvzirqlzpylxxij8q10hramdsgk6nvx-AMB-plugins-0.8.1
/nix/store/szh7aikwz12vj1sbkf4r6vdvy1k8apym-archisteamfarm-5.1.5.3
/nix/store/mxi5pignri1z8n3lizkcp8y8m8cgfn55-AusweisApp2-1.22.2
/nix/store/6d69d3lbk6dbgqnvzccx9lpi7hj3f6i9-CHOWTapeModel-2.10.0
/nix/store/lhk99h3adbfhzdk1spcx7awky7bzhwab-ChowKick-1.1.1
/nix/store/6p5g08rrq4cf5yrs970h4qrv7drj812l-CoinMP-1.8.4
/nix/store/qlxs8yrmssi1x54a5m3q0941ha4qvaa8-coreaction-4.2.0
/nix/store/64r6kkd6cxm1jhlhl9zma80jgvidrfyw-corearchiver-4.2.0
/nix/store/mh79zi4l343fj1xa54p1a4qif38f4435-corefm-4.2.0
/nix/store/6disqp58dkyi8i3vpb6ja5sr8g4xpqxg-coregarage-4.2.0
โ ๏ธ I could not seem to make this parallel at all since the evaluation requires a global lock.
Now I just need to rename outpaths.txt to something that indicates the Git commit I used when generating this and I am starting to build some nice structure data ๐.
Of course what I am doing here is only generating for my current builtins.currentSystem (i.e. x86_64) and will not generate for all other supported platforms.
There are also recursive subtrees within nixpkgs such as pkgsStatic or pkgsMusl that I donโt believe are returned from nix search and therefore I am not detecting those output paths.
I would like to continue to understand better how tools such as Hydra generate all attributes for all systems.
If this problem and goal to build an outputPath reverse index for Nix sounds interesting, reach out to me! I would love to collaborate.
]]>Our CI infrastructure leverages containers (donโt they all now?) for each job and we explored different solutions to reduce the cost of constantly downloading the /nix/store necessary for the build.
At first, we implemented a solution where the /nix/store was embedded within the Docker image itself.
This was an interesting choice for a few reasons:
We decided to seek out alternatives where we can remove the prebuild /nix/store from the container but still reduce the cold-boot cost for CI jobs.
The caveat to the solution we seeked is that the Docker image still installed Nix which meant it created a /nix/store entry and the necessary ~/.nix_profile symlinks.
Overlayfs allows one, usually read-write, directory tree to be overlaid onto another, read-only directory tree. All modifications go to the upper, writable layer.
Letโs explore how we might go about setting this up! Letโs use a dummy example.
FROM ubuntu
# Let's make a dummy nix-store
RUN mkdir -p /nix/store
# let's put a dummy derivation
RUN echo "hello" > /nix/store/hello
# a dummy command
CMD ["/bin/bash"]
I will also create some dummy directories on my host.
mkdir -p /tmp/fake-nix/{upper,workdir}
echo "ping" > /tmp/fake-nix/upper/pong
Letโs spin up a docker container.
# lets run a docker and bind-mount of host /nix/store
# in this case we called it /tmp/fake-nix
# In reality this will be your host's /nix/store
docker run --privileged -v /tmp/fake-nix:/nix/store-host \
-it $(docker build . --quiet) /bin/bash
Letโs check the contents of the /nix/store originally. We see that it only has out hello file.
root@c32024e56f25:/# ls /nix/store
hello
Now letโs mount our overlay filesystem.
root@c32024e56f25:/# mount -t overlay overlay -o \
lowerdir=/nix/store,upperdir=/nix/store-host/upper,workdir=/nix/store-host/workdir \
/nix/store
Letโs check the contents of our /nix/store now.
root@c32024e56f25:/# ls -l /nix/store
total 8
-rw-r--r-- 1 root root 6 Sep 9 17:15 hello
-rw-r--r-- 1 780412 89939 5 Sep 10 16:22 pong
Great! Our /nix/store now has the contents of the host overlaid ontop of the one within the container.
What if we write a new file?
root@c32024e56f25:/# echo "test" > /nix/store/test
root@c32024e56f25:/# ls /nix/store
hello pong test
root@c32024e56f25:/# ls /nix/store-host/upper/
pong test
We see that it created the file in the upper directory.
What if we update a file in the lower directory?
root@c32024e56f25:/# echo "world" > /nix/store/hello
root@c32024e56f25:/# ls /nix/store-host/upper/
hello pong test
Ok cool โ it moved it to our upper directory.
What if we delete the file?
root@c32024e56f25:/# rm /nix/store/hello
root@c32024e56f25:/# ls /nix/store
pong test
root@c32024e56f25:/# ls -l /nix/store-host/upper
total 8
c--------- 2 root root 0, 0 Sep 11 02:22 hello
-rw-r--r-- 1 780412 89939 5 Sep 10 16:22 pong
-rw-r--r-- 1 root root 5 Sep 11 02:19 test
The hello file still exists in the upper directory but itโs a tombstone file now. This is to distinguish it from the fact itโs been deleted while still existing in the lower directory.
Awesome โ whatโs great too is that the overlay filesystem thatโs supported natively in Linux gives you near native performance.
The implementation differs from other โunion filesystemโ implementations in that after a file is opened all operations go directly to the underlying, lower or upper, filesystems. This simplifies the implementation and allows native performance in these cases.
Hope that helps others in trying to speedup their builds. ๐ค
]]>Similar to the movies I watch that take place in space, it only takes a small hole to destroy your hermetic environment. ๐งโ๐
Iโve written previously about my encounters trying to remove all impurities within a JVM environment.
Iโve actually upstreamed fixing the default java.library.path for the OpenJDK distributed by Nixpkgs. ๐
Awesome! That should have solved my problem, right ?โฆ
Unfortunately, impurities are tough to stamp out. NixOS is trying to accomplish a paradigm shift by dismissing the filesystem hierarchy standard but it is deeply rooted in assumptions when people build for Linux.
In my specific case, the search paths for the JNR libraries were hardcoded.
// paths should have no duplicate entries and have insertion order
LinkedHashSet<String> paths = new LinkedHashSet<String>();
try {
paths.addAll(getPropertyPaths("jnr.ffi.library.path"));
paths.addAll(getPropertyPaths("jaffl.library.path"));
// Add JNA paths for compatibility
paths.addAll(getPropertyPaths("jna.library.path"));
// java.library.path should take care of Windows defaults
paths.addAll(getPropertyPaths("java.library.path"));
} catch (Exception ignored) {
}
if (Platform.getNativePlatform().isUnix()) {
// order is intentional!
paths.add("/usr/local/lib");
paths.add("/usr/lib");
paths.add("/lib");
}
Iโve sent a fix upstream please feel free to comment.
This meant that I had to make sure the java.library.path resolved to the glibc I am using in Nixpkgs first.
I couldnโt get rid of the JAVA_TOOL_OPTIONS
just yet. ๐ค
First off, the JDK emits JAVA_TOOL_OPTIONS
via stderr which is non-configurable.
if (os::getenv(name, buffer, sizeof(buffer)) &&
!os::have_special_privileges()) {
JavaVMOption options[N_MAX_OPTIONS]; // Construct option array
jio_fprintf(defaultStream::error_stream(),
"Picked up %s: %s\n", name, buffer);
This is frustrating because plenty of tools (i.e. IntelliJ) assume failure if anything is emitted to stderr.
Secondly, we have many developers that are being hit by this particular workflow:
If you read my earlier post, youโll understand why JAVA_TOOL_OPTIONS includes glibc.
That JAVA_TOOL_OPTIONS references glibc Z but their JRuby SDK is pointing to JRuby A which was built against glibc B.
They are then graced with this wonderful message.
java.lang.UnsatisfiedLinkError: /nix/store/cvr0kjg2q7z2wwhjblx6c73rv422k8cm-glibc-2.33-47/lib/libc.so.6: undefined symbol: _dl_catch_error_ptr, version GLIBC_PRIVATE
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:93)
Ugh! The fix here is straightforward ultimately โ the developer needs to be mindful of their JRuby SDK set in IntelliJ and keep it in sync with their local checkout.
Unfortunately itโs a sharp edge many are running into; and I donโt blame them!
We will be thinking of some sane ways to keep these two tools in sync so that we can remain on upstream for nowโฆ ๐ค
]]>