About me: My name is Solène Rapenne, pronouns she/her. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD Qubes OS Lisp cmdline gaming security QubesOS internet-stuff). I love percent and lambda characters. OpenBSD developer solene@. No AI is involved in this blog.
Contact me: solene at dataswamp dot org or @solene@bsd.network (mastodon).
You can sponsor my work financially if you want to help me writing this blog and contributing to Free Software as my daily job.
This may appear like a very niche use case, in my quest of software conservancy for nixpkgs I didn't encounter many people understanding why I was doing this.
I would like to present you a project I made to easily download all the sources files required to build packages from nixpkgs, allowing to keep offline copies.
Why would you like to keep a local copy? If upstream disappear, you can't get access to the sources anymore, except maybe in Hydra, but you rely on a third party to access the sources, so it's still valuable to have local copies of software you care about, just to make copies. It's not that absolutely useful for everyone, but it's always important to have such tools available.
nixpkgs-mirror-tarballs project page
You must run it on a system with nix installed.
After cloning and 'cd-ing' into the directory, simply run ./run.sh some package list | ./mirror.pl. The command run.sh will generate a JSON structure containing all the dependencies used by the packages listed as arguments, and the script mirror.pl will iterate over the JSON list and use nix's fetcher to gather the sources in the nix store, verifying the checksum on the go. This will create a directory distfiles containing symlinks to the sources files stored in the store.
The symlinks are very important as they will prevent garbage collection from the store, and it's also used internally to quickly check if a file is already in the store.
To delete a file from the store, remove its symlink and run the garbage collector.
I still need to figure how to get a full list of all the packages, I currently have a work in progress relying on nix search --json but it doesn't work on 100% of the packages for some reasons.
It's currently not possible to easily trim distfiles that aren't useful anymore, I plan to maybe add it someday.
This task is natively supported in the OpenBSD tool building packages (dpb), it can fetch multiples files in parallel and automatic remove files that aren't used anymore. This was really complicated to figure how to replicate this with nixpkgs.