About me: My name is Solène Rapenne, pronouns she/her. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD Qubes OS Lisp cmdline gaming security QubesOS internet-stuff). I love percent and lambda characters. OpenBSD developer solene@. No AI is involved in this blog.
Contact me: solene at dataswamp dot org or @solene@bsd.network (mastodon).
You can sponsor my work financially if you want to help me writing this blog and contributing to Free Software as my daily job.
Written by Solène, on 31 October 2022.
Tags:
#openbsd
#nixos
#nushell
#shell
Let me introduce you to a nice project I found while lurking on the Internet. It's called nushell and is a non-POSIX shell, so most of your regular shells knowledge (zsh, bash, ksh, etc…) can't be applied on it, and using it feels like doing functional programming.
It's a good tool for creating robust data manipulation pipelines, you can think of it like a mix of a shell which would include awk's power, behave like a SQL database, and which knows how to import/export XML/JSON/YAML/TOML natively.
You may want to try nushell only as a tool, and not as your main shell, it's perfectly fine.
With a regular shell, iterating over a command output can be complex when it involves spaces or newlines, for instance, that's why find and xargs have a -print0 parameter to have a special delimited between "items", but it doesn't compose well with other tools. Nushell handles correctly this situation as its manipulates the data using indexed entries, given you correctly parsed the input at the beginning.
Nushell is a rust program, so it should work on every platform where Rust/Cargo are supported. I packaged it for OpenBSD, so it's available on -current (and will be in releases after 7.3 is out), the port could be used on 7.2 with no effort.
With Nix, it's packaged under the name nushell, the binary name is nu.
For other platforms, it's certainly already packaged, otherwise you can find installation instructions to build it from sources.
Nushell documentation: Building nushell from sources
At first run, you are prompted to use default configuration files, I'd recommend accepting, you will have files created in ~/.config/nushell/.
The only change I made from now is to make Tab completion case-sensitive, so D[TAB] completes to Downloads instead of asking between dev and Downloads. Look for case_sensitive_completions in .config/nushell/config.nu and set it to true.
If you are like me, and you prefer learning by doing instead of reading a lot of documentation, I prepared a bunch of real world use case you can experiment with. The documentation is still required to learn the many commands and syntax, but examples are a nice introduction.
Help from nushell can be parsed directly with nu commands, it's important to understand where to find information about commands.
Use help a-command to learn from a single command:
> help help
Display help information about commands.
Usage:
> help {flags} ...(rest)
Flags:
-h, --help - Display this help message
-f, --find <String> - string to find in command names, usage, and search terms
[cut so it's not too long]
Use help commands to list all available commands (I'm limiting to 5 between there are a lot of commands)
help commands | last 5
╭───┬─────────────┬────────────────────────┬───────────┬───────────┬────────────┬───────────────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ # │ name │ category │ is_plugin │ is_custom │ is_keyword │ usage │ search_terms │
├───┼─────────────┼────────────────────────┼───────────┼───────────┼────────────┼───────────────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ 0 │ window │ filters │ false │ false │ false │ Creates a sliding window of `window_size` that slide by n rows/elements across input. │ │
│ 1 │ with-column │ dataframe or lazyframe │ false │ false │ false │ Adds a series to the dataframe │ │
│ 2 │ with-env │ env │ false │ false │ false │ Runs a block with an environment variable set. │ │
│ 3 │ wrap │ filters │ false │ false │ false │ Wrap the value into a column. │ │
│ 4 │ zip │ filters │ false │ false │ false │ Combine a stream with the input │ │
╰───┴─────────────┴────────────────────────┴───────────┴───────────┴────────────┴───────────────────────────────────────────────────────────────────────────────────────┴──────────────╯
Add sort-by category to list them... sorted by category.
help commands | sort-by category
Use where category == filters to only list commands from the filters category.
help commands | where category == filters
Use find foobar to return lines containing foobar.
help commands | find insert
This is just an example from YAML to JSON, but you can convert much more formats into other formats.
open dev/home-impermanence/tests/impermanence.yml | to json
{
"directories":
[
"Documents",
"Downloads",
"Datastore/Music",
"Datastore",
"Datastore/",
"Datastore/Music/Band1",
".config",
"foo/bar",
"foo/bar/hello"
],
"size": "500m",
"files":
[
".Xdefaults",
".profile",
".xsession",
]
}
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)"
Because the output would be too long, here is how you get 10 random keys from sysctl.
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" | shuffle | last 10 | sort-by key
╭───┬─────────────────────────────────────────────────┬──────────╮
│ # │ key │ value │
├───┼─────────────────────────────────────────────────┼──────────┤
│ 0 │ fs.quota.reads │ 0 │
│ 1 │ net.core.high_order_alloc_disable │ 0 │
│ 2 │ net.ipv4.conf.all.drop_gratuitous_arp │ 0 │
│ 3 │ net.ipv4.conf.default.rp_filter │ 2 │
│ 4 │ net.ipv4.conf.lo.disable_xfrm │ 1 │
│ 5 │ net.ipv4.conf.lo.forwarding │ 0 │
│ 6 │ net.ipv4.ipfrag_low_thresh │ 3145728 │
│ 7 │ net.ipv6.conf.all.ioam6_id │ 65535 │
│ 8 │ net.ipv6.conf.all.router_solicitation_interval │ 4 │
│ 9 │ net.mptcp.enabled │ 1 │
╰───┴─────────────────────────────────────────────────┴──────────╯
A complicated task using a regular shell, recursively find files matching a pattern and then run a given command on each of them, in parallel. Which is exactly what you need if you want to convert your music library into another format, let's convert everything from FLAC to OPUS in this example.
In the following command line, we will look for every .flac file in the subdirectories, then run in parallel using par-each the command ffmpeg on it, from its current name to the old name with .flac changed to .opus.
The let convert and | complete commands are used to store the output of each command into a result table, and store it in the variable convert so we can query it after the job is done.
let convert = (ls **/*flac | par-each { |file| do -i { ffmpeg -i $file.name ($file.name | str replace flac opus) } | complete })
Now, we have a structure in convert that contains the columns stdout, stderr and exit_code, so we can look if all the commands did run correctly using the following query.
$convert | where exit_code != 0
I had a special need for my phone and my huge music library, I wanted to have a lower quality version of it synced with syncthing, but I needed this to be easy to update when adding new files.
It takes all the music files in /home/user/Music/ and creates a 64K opus file in /home/user/Stream/ by keeping the same file tree hierarchy, and if the opus destination file exists it's skipped.
cd /home/user/Music/
let dest = "/home/user/Stream/"
let convert = (ls **/* |
where name =~ ".(mp3|flac|opus|ogg)$" |
where name !~ "(Audiobook|Piano)" |
par-each {
|file| do -i {
let new_name = ($file.name | str replace -r ".(flac|ogg|mp3)" ".opus")
if (not ([$dest, $new_name] | str join | path exists)) {
mkdir ([$dest, ($file.name | path dirname)] | str join)
ffmpeg -i $file.name -b:a 64K ([$dest, $new_name] | str join)
} | complete
}
})
$convert
I have a lot of digitalized books/mangas/comics, this conversion is a handy operation reducing the size of the files by 40% (up to 70%).
def conv [] {
if (ls | first | get name | str contains ".jpg") {
ls *jpg | par-each {|file| do -i { cwebp $file.name -o ($file.name | str replace jpg webp) } | complete }
rm *jpg
}
if (ls | first | get name | str contains ".ppm") {
ls *ppm | par-each {|file| do -i { cwebp $file.name -o ($file.name | str replace ppm webp) } | complete }
rm *ppm
}
}
ls * | each {|file| do -i {
if ($file.name | str contains ".cbz") { unzip $file.name -d ./pages/ } ;
if ($file.name | str contains ".cbr") { unrar e -op./pages/ $file.name } ;
if ($file.name | str contains ".pdf") { mkdir pages ; pdfimages $file.name pages/page } ;
cd pages ; conv ; cd ../ ; ^zip -r $"($file.name).webp.cbz" pages ; rm -fr pages
} }
〉tar vtf nushell.tgz | parse -r "(.*?) (.*?)\/(.*?)\\s+(.*?) (.*?) (.*?) (.*)" | rename mode owner group size date time path
╭───┬────────────┬────────┬───────┬───────┬────────────┬───────┬────────────────────╮
│ # │ mode │ owner │ group │ size │ date │ time │ path │
├───┼────────────┼────────┼───────┼───────┼────────────┼───────┼────────────────────┤
│ 0 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 16:45 │ nushell │
│ 1 │ -rw-r--r-- │ solene │ wheel │ 519 │ 2022-10-30 │ 13:41 │ nushell/Makefile │
│ 2 │ -rw-r--r-- │ solene │ wheel │ 29304 │ 2022-10-29 │ 18:49 │ nushell/crates.inc │
│ 3 │ -rw-r--r-- │ solene │ wheel │ 75003 │ 2022-10-29 │ 13:16 │ nushell/distinfo │
│ 4 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 00:00 │ nushell/pkg │
│ 5 │ -rw-r--r-- │ solene │ wheel │ 337 │ 2022-10-29 │ 18:52 │ nushell/pkg/DESCR │
│ 6 │ -rw-r--r-- │ solene │ wheel │ 14 │ 2022-10-29 │ 18:53 │ nushell/pkg/PLIST │
╰───┴────────────┴────────┴───────┴───────┴────────────┴───────┴────────────────────╯
〉open --raw freq.ods | from ods | get Sheet1 | headers
╭───┬─────────────┬──────────────┬───────────┬─────────┬───────────────┬────────────┬───────┬─────────┬─────────┬──────────╮
│ # │ Policy │ Compile time │ Idle time │ column3 │ Compile power │ Idle power │ Total │ column8 │ column9 │ column10 │
├───┼─────────────┼──────────────┼───────────┼─────────┼───────────────┼────────────┼───────┼─────────┼─────────┼──────────┤
│ 0 │ powersaving │ 1123.00 │ 0.00 │ │ 5.90 │ 0.00 │ 5.90 │ │ │ │
│ 1 │ auto │ 871.00 │ 252.00 │ │ 5.60 │ 0.74 │ 6.34 │ │ 0.44 │ 6.94 │
╰───┴─────────────┴──────────────┴───────────┴─────────┴───────────────┴────────────┴───────┴─────────┴─────────┴──────────╯
We can format new strings from columns values.
〉open --raw freq.ods | from ods | get Sheet1 | headers | each {|row| do { echo $"($row.Policy) = ($row.'Compile power' + $row.'Idle power') Watts" } }
╭───┬─────────────────────────╮
│ 0 │ powersaving = 5.9 Watts │
│ 1 │ auto = 6.34 Watts │
╰───┴─────────────────────────╯
There is a website listing packages that can be updated on OpenBSD at https://portroach.openbsd.org, it provides json of data for rendering.
We can use this data to sort which maintainer has the most up to date percentage, but only if they manage more than 30 packages.
fetch https://portroach.openbsd.org/json/totals.json | get results | where total > 30 | sort-by percentage
nix profile list | parse "{index} {flake} {source} {store}"
╭───┬───────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────╮
│ # │ flake │ source │ store │
├───┼───────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 0 │ flake:nixpkgs#legacyPackages.x86_64-linux.libreoffice │ path:/nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source?lastModified=1663494472& │ /nix/store/1m6wp1pznhf2nrvs7xwmvig5x3nspq0j-libreoffice-7.2.6.2 │
│ │ │ narHash=sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb%2ftFi%2fqM=&rev=f677051b8dc0 │ │
│ │ │ b5e2a9348941c99eea8c4b0ff28f#legacyPackages.x86_64-linux.libreoffice │ │
│ 1 │ flake:nixpkgs#legacyPackages.x86_64-linux.dino │ path:/nix/store/9cj1830pvd88lrwmmxw65achd3lw2q9n-source?lastModified=1667050928& │ /nix/store/ljhn4n1q5pk7wr337v681m1h39jp5l2y-dino-0.3.0 │
│ │ │ narHash=sha256-xOn0ZgjImIyeecEsrjxuvlW7IW5genTwvvnDQRFncB8=&rev=fdebb81f45a1ba2c │ │
│ │ │ 4afca5fd9f526e1653ad0949#legacyPackages.x86_64-linux.dino │ │
╰───┴───────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────╯
nix flake show --json | from json
╭────────────────┬───────────────────╮
│ defaultPackage │ {record 5 fields} │
│ packages │ {record 5 fields} │
╰────────────────┴───────────────────╯
nix flake show --json | from json | get packages
╭────────────────┬───────────────────╮
│ aarch64-darwin │ {record 2 fields} │
│ aarch64-linux │ {record 2 fields} │
│ i686-linux │ {record 2 fields} │
│ x86_64-darwin │ {record 2 fields} │
│ x86_64-linux │ {record 2 fields} │
╰────────────────┴───────────────────╯
nix flake show --json | from json | get packages.x86_64-linux
╭───────────────┬───────────────────╮
│ nix-dev-html │ {record 2 fields} │
│ nix-dev-pyenv │ {record 3 fields} │
╰───────────────┴───────────────────╯
> open flake.lock | from json | get nodes.nixpkgs.locked
╭──────────────┬─────────────────────────────────────────────────────╮
│ lastModified │ 1663494472 │
│ narHash │ sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb/tFi/qM= │
│ path │ /nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source │
│ rev │ f677051b8dc0b5e2a9348941c99eea8c4b0ff28f │
│ type │ path │
╰──────────────┴─────────────────────────────────────────────────────╯
> open /etc/fstab | from ssv -m 1 -n | rename device mountpoint fs options freq passno
_────┬────────────────────┬─────────────────┬──────┬───────────────────────────────────────────┬──────┬────────_
│ # │ device │ mountpoint │ fs │ options │ freq │ passno │
├────┼────────────────────┼─────────────────┼──────┼───────────────────────────────────────────┼──────┼────────┤
│ 0 │ 55a6c21017f858cb.b │ none │ swap │ sw │ __ │ __ │
│ 1 │ 55a6c21017f858cb.a │ / │ ffs │ rw,noatime,softdep │ 1 │ 1 │
│ 2 │ 55a6c21017f858cb.l │ /home │ ffs │ rw,noatime,wxallowed,softdep,nodev,nosuid │ 1 │ 2 │
│ 3 │ 55a6c21017f858cb.d │ /tmp │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 4 │ 55a6c21017f858cb.f │ /usr │ ffs │ rw,noatime,softdep,nodev │ 1 │ 2 │
│ 5 │ 55a6c21017f858cb.g │ /usr/X11R6 │ ffs │ rw,noatime,softdep,nodev │ 1 │ 2 │
│ 6 │ 55a6c21017f858cb.h │ /usr/local │ ffs │ rw,noatime,softdep,wxallowed,nodev │ 1 │ 2 │
│ 7 │ 55a6c21017f858cb.k │ /usr/obj │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 8 │ 55a6c21017f858cb.j │ /usr/src │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 9 │ 55a6c21017f858cb.e │ /var │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 10 │ afebb2a83a449265.b │ /build │ ffs │ rw,noatime,softdep,wxallowed,nosuid │ 1 │ 2 │
│ 11 │ afebb2a83a449265.a │ /build/pobj │ ffs │ rw,noatime,softdep,nodev,wxallowed,nosuid │ 1 │ 2 │
│ 12 │ 55a6c21017f858cb.b │ /build/pobj_mfs │ mfs │ -s1G,wxallowed,noatime,rw │ 0 │ 0 │
╰────┴────────────────────┴─────────────────┴──────┴───────────────────────────────────────────┴──────┴────────_
open /var/log/messages | parse -r "(?<date>\\w+ \\d+ \\d+:\\d+:\\d+) (?<hostname>\\w+) (?<program>\\w+)\\[?(?<pid>\\d+)?\\]?: (?<message>.*)"
╭───┬─────────────────┬──────────┬────────────┬───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ # │ date │ hostname │ program │ pid │ message │
├───┼─────────────────┼──────────┼────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 0 │ Oct 31 10:27:32 │ fx6 │ collectd │ 55258 │ uc_update: Value too old: name = fx6openbsd/swap/swap-free; value time = 1667208452.108; last cache update = 1667208452.108; │
│ 1 │ Oct 31 10:43:02 │ fx6 │ collectd │ 55258 │ uc_update: Value too old: name = fx6openbsd/swap/percent-free; value time = 1667209382.102; last cache update = 1667209382.102; │
│ 2 │ Oct 31 11:00:01 │ fx6 │ syslogd │ 4629 │ restart │
│ 3 │ Oct 31 11:05:26 │ fx6 │ pkg_delete │ │ Removed helix-22.08.1 │
│ 4 │ Oct 31 11:05:29 │ fx6 │ pkg_add │ │ Added helix-22.08.1 │
│ 5 │ Oct 31 11:16:49 │ fx6 │ pkg_add │ │ Added llvm-13.0.0p3 │
│ 6 │ Oct 31 11:20:18 │ fx6 │ pkg_add │ │ Added clang-tools-extra-13.0.0p2 │
│ 7 │ Oct 31 11:20:32 │ fx6 │ pkg_add │ │ Added bash-5.2.2 │
│ 8 │ Oct 31 11:20:34 │ fx6 │ pkg_add │ │ Added fzf-0.34.0 │
│ 9 │ Oct 31 11:21:01 │ fx6 │ pkg_delete │ │ Removed fzf-0.34.0 │
╰───┴─────────────────┴──────────┴────────────┴───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
pkg_info | str trim | parse -r "(?<package>.*?)-(?<version>[a-zA-Z0-9\\.]*?) (?<description>.*)" | str trim description
╭────┬───────────────────┬────────────┬────────────────────────────────────────────────────╮
│ # │ package │ version │ description │
├────┼───────────────────┼────────────┼────────────────────────────────────────────────────┤
│ 0 │ athn-firmware │ 1.1p4 │ firmware binary images for athn(4) driver │
│ 1 │ collectd │ 5.12.0 │ system metrics collection engine │
│ 2 │ curl │ 7.85.0 │ transfer files with FTP, HTTP, HTTPS, etc. │
│ 3 │ gettext-runtime │ 0.21p1 │ GNU gettext runtime libraries and programs │
│ 4 │ intel-firmware │ 20220809v0 │ microcode update binaries for Intel CPUs │
│ 5 │ inteldrm-firmware │ 20220913 │ firmware binary images for inteldrm(4) driver │
│ 6 │ kakoune │ 2021.11.08 │ modal code editor with a focus on interactivity │
│ 7 │ libgcrypt │ 1.10.1p0 │ crypto library based on code used in GnuPG │
│ 8 │ libgpg-error │ 1.46 │ error codes for GnuPG related software │
│ 9 │ libiconv │ 1.17 │ character set conversion library │
│ 10 │ libstatgrab │ 0.91p5 │ system statistics gathering library │
│ 11 │ libxml │ 2.10.3 │ XML parsing library │
│ 12 │ libyajl │ 2.1.0 │ small JSON library written in ANSI C │
│ 13 │ nghttp2 │ 1.50.0 │ library for HTTP/2 │
│ 14 │ nushell │ 0.70.0 │ a new kind of shell │
│ 15 │ obsdfreqd │ 1.0.3 │ userland daemon to manage CPU frequency │
│ 16 │ quirks │ 6.42 │ exceptions to pkg_add rules and cache │
│ 17 │ rsync │ 3.2.5pl0 │ mirroring/synchronization over low bandwidth links │
│ 18 │ ttyplot │ 1.4p0 │ realtime plotting utility for terminals │
│ 19 │ vmm-firmware │ 1.14.0p0 │ firmware binary images for vmm(4) driver │
│ 20 │ xz │ 5.2.7 │ LZMA compression and decompression tools │
│ 21 │ yash │ 2.52 │ POSIX-compliant command line shell │
╰────┴───────────────────┴────────────┴────────────────────────────────────────────────────╯
Nushell is very fun, it's terribly different from regular shells, but it comes with a powerful language and tooling. I always liked shells because of pipes commands, allowing to construct a complex transformation/analysis step by step, and easily inspect any step, or be able to replace a step by another.
With nushell, it feels like I finally have a better tool to create more reliable, robust, portable and faster command pipelines. The learning curve didn't feel too hard, but maybe it's because I'm already used to functional programming.