Modern utilities

Author

Marie-Hélène Burle

In recent years, a number of open-source utilities for the Unix shell have emerged. Some are meant as replacements for classic tools with improved performance, better defaults, or nicer-looking outputs; others add novel functionality. Several of them were recently installed on the Alliance clusters.

In this section we will cover a selection of tools that are very popular, well-maintained, and will improve your time on the command line.

How to choose tools?

You can install and experiment with any tool. A few things might help you decide whether or not a tool looks promising:

How popular is it? (GitHub stars)
Is it maintained? (date of last commit)
How polished is the documentation?
How fast is it? (what language is it written in?)

Tools written in Shell or Python will be slower, while those written in compiled languages (Rust, C, Go) will be faster.

`ls` in colours: `eza`

eza is a replacement for ls.

For the most part, it adds colours and uses the same options as ls, but it also adds a few features and has better default options.

Installation

On your own machine

You can find the installation instructions here.

On the Alliance clusters

eza is not installed on the Alliance clusters, so you will have to install it locally under your own user. This is easy to do because it is written in Rust and can be installed with the Rust package manager.

First, you load a Rust module, then you install the package:

module load rust/1.76.0
cargo install eza

Finally, make sure that ~/.cargo/bin is in your PATH as this is where the binary will get installed. So add to your .bashrc:

export PATH=$PATH:~/.cargo/bin

You only need to do this once in the production cluster you use. After eza has been installed, it will be accessible on subsequent sessions.

Usage

eza

projects scratch

You now have different colours for different types of files. You can already see that the directory (projects) is in bold blue and the symlink (scratch) is in cyan (my website doesn’t render these colours but you will see them when you run the command in our training cluster).

Let’s create some files quickly to see a bit more colours:

touch test.html test.md test.py test.R
eza

projects scratch test.html test.md test.py test.R

Now you can see that the scripts are in bold yellow while the markup files are in gray.

The flags are similar to those of ls, but some defaults are slightly different:

eza -al

.rw-------@ 2.8k user02 19 Dec 04:20 .bash_history
.rw-------@   18 user02 13 Dec 00:55 .bash_logout
.rw-r-----@  141 user02 19 Dec 04:20 .bash_profile
.rw-r-----@ 7.3k user02 19 Dec 03:57 .bashrc
drwxr-x---@    - user02 19 Dec 03:57 .cargo
drwx------@    - user02 13 Dec 18:10 .ssh
drwxr-xr-x@    - root   13 Dec 00:57 projects
lrwxrwxrwx@    - user02 13 Dec 00:55 scratch -> /scratch/user02
.rw-r-----@    0 user02 19 Dec 04:25 test.html
.rw-r-----@    0 user02 19 Dec 04:25 test.md
.rw-r-----@    0 user02 19 Dec 04:25 test.py
.rw-r-----@    0 user02 19 Dec 04:29 test.R

Compare with ls -al:

total 60
drwx------.  12 user02 user02 4096 Dec 19 04:29 .
drwxr-xr-x. 101 root   root   4096 Dec 13 00:57 ..
-rw-------.   1 user02 user02 2763 Dec 19 04:20 .bash_history
-rw-------.   1 user02 user02   18 Dec 13 00:55 .bash_logout
-rw-r-----.   1 user02 user02  141 Dec 19 04:20 .bash_profile
-rw-r-----.   1 user02 user02 7320 Dec 19 03:57 .bashrc
drwxr-x---.   4 user02 user02  125 Dec 19 03:57 .cargo
drwx------.   2 user02 user02   48 Dec 13 18:10 .ssh
drwxr-xr-x.   2 root   user02   27 Dec 13 00:57 projects
lrwxrwxrwx.   1 user02 user02   15 Dec 13 00:55 scratch -> /scratch/user02
-rw-r-----.   1 user02 user02    0 Dec 19 04:29 test.R
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.html
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.md
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.py

eza by default shows the output in a human readable format (-h flag with ls) and without the group (-G with ls).

There is also the additional -T flag which is equivalent to running the tree utility:

eza -T

.
├── projects
│   └── def-sponsor00 -> /project/def-sponsor00
├── scratch -> /scratch/user02
├── test.html
├── test.md
├── test.py
└── test.R

Alias

If you want, you can alias it to ls by adding to your .bashrc file:

alias ls=eza

If you ever want to use the true ls utility, you can do so with \ls.

Alternative

If you want a simpler and more lightweight way to add colours to your ls outputs, you can look at LS_COLORS. I did this for years until I found eza.

To install it locally in the Alliance clusters, you have to download and uncompress a script, and copy it to a proper location:

mkdir ./LS_COLORS &&
    curl -L https://api.github.com/repos/trapd00r/LS_COLORS/tarball/master | tar xzf - --directory=./LS_COLORS --strip=1 &&
    mkdir -p ~/.local/share &&
    cp ~/LS_COLORS/lscolors.sh ~/.local/share &&
    rm -r ~/LS_COLORS

Then you need to add the following to your .bashrc file to source the script and add a proper alias to ls:

source ~/.local/share/lscolors.sh
alias ls='ls --color'

A `cat` with wings: `bat`

bat is a replacement for cat with syntax highlighting for most programming languages, line numbers, and pager-like search and navigation.

Installation

bat is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

Use bat as you would use cat:

bat .bash_profile

───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .bash_profile
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ # .bash_profile
   2   │
   3   │ # Get the aliases and functions
   4   │ if [ -f ~/.bashrc ]; then
   5   │     . ~/.bashrc
   6   │ fi
   7   │
   8   │ # User specific environment and startup programs
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────

Then you are in your default pager.

Among other options, you can disable the frame with -nand also remove the line numbers with -p.

Faster `find`: `fd`

fd is a replacement for find with the following assets:

written in Rust, automatic parallelism ➔ with vastly improved performance,
more friendly syntax,
by default excludes binaries as well as hidden files and directories,
by default excludes patterns from .gitignore or other .ignore files.

Installation

fg is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Basic usage

Search file names for a pattern recursively in current directory:

fd te

test.R
test.html
test.md
test.py

fd uses regexp by default, so you can use pattern symbols:

fd jx.*txt

Search file names recursively in another directory:

fd top bash/

Print all files in some directory to stdout

Current directory:

fd

Another directory:

fd . bash/

Options

Search for files with a particular file extension:

fd -e txt

Use a globbing pattern instead of regexp:

fd -g wb* bash/

Match full path instead of simply file name:

fd -p bash/wb

Execute command for each result of fd in parallel:

fd top bash/ -x rg layout

Execute command once with all results of fd as arguments:

fd top bash/ -X rg layout

Excluded files and directories

By default, fd excludes hidden files/directories and patterns in .gitignore (you can disable this with -H and -I respectively).

This makes fd combined with tree sometimes more useful than tree alone:

fd . bash/ | tree --fromfile

You can make this a function:

ft () { fd $@ | tree --fromfile }

Exclude additional directories or patterns:

fd -E *.txt -E img/ . bash/

I personally prefer to disable the default settings and exclude patterns based on a file I created:

alias fd='fd -u --ignore-file /home/marie/.fdignore'

This file (which can have any name you want) uses the same syntax as .gitignore.

RIP `grep`: ripgrep

ripgrep provides the rg utility—a replacement for grep with the following assets:

written in Rust, automatic parallelism ➔ with vastly improved performance,
by default excludes patterns from .gitignore or other .ignore files,
by default excludes binaries as well as hidden files and directories,
by default doesn’t follow symlinks.

Installation

rg is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

Search lines in a file matching a pattern:

rg .bash .bash_profile

1:# .bash_profile
4:if [ -f ~/.bashrc ]; then
5:  . ~/.bashrc

Search lines matching pattern in all files in current directory (recursively):

rg .bash

rg and fd follow the same principles:

use of regexp by default,
use of globbing pattern with -g,
search recursively by default,
same excluded files.

You can find full details on the syntax here.

Smart `cd`: `zoxide`

zoxide allows to easily jump to any directory.

Installation

Instructions to install zoxide on your machine can be found here. Note that fzf (see below) adds cool functionality to it, so you might want to install it as well.

Installing zoxide on the Alliance clusters is extremely easy.

First, you have to run the command:

curl -sSfL https://raw.githubusercontent.com/ajeetdsouza/zoxide/main/install.sh | sh

This will install the binary in ~/.local/bin.

Then, you have to make sure that ~/.local/bin is in your PATH.

Add to your .bashrc:

export PATH=$PATH:~/.local/bin

Finally, you need to add to your .bashrc file (if you want to use it in Bash):

eval "$(zoxide init bash)"

Or, if you want to use it in Zsh, add to your .zshrc file:

eval "$(zoxide init zsh)"

You can change the z and zi commands (a bit hard to type) by whatever command you want. I personally changed them to j and ji (“j” for “jump”). To do this, replace the eval expression(s) above by:

eval "$(zoxide init --cmd j bash)"

and/or

eval "$(zoxide init --cmd j zsh)"

Usage

Now, you can type z (or whatever command you chose—see above) instead of a regular cd command.

In addition, you can simplify the path you want to get to to just a few characters and zoxide will take you there.

If there are multiple locations matching your entry, the matching algorithm will chose the highest ranking path based on your frequency of use and how recently you visited a path.

This means that you can visit your usual places extremely easily. For less frequent places, add a bit more info.

Finally, if you want to choose amongst all possible options in a completion framework, use zi instead and zoxide will open fzf (see below).

Alternative

A tool that served me well until someone pointed the faster and better zoxide to me is autojump.

Installation

autojump is installed on the Alliance clusters, but you need to source a script before you can use it. Since you need to do this at every session, you should add to your .bashrc file:

[[ -s $EPREFIX/etc/profile.d/autojump.sh ]] && source $EPREFIX/etc/profile.d/autojump.sh

To install autojump on your machine, you can follow the instructions here.

Usage

Similar to zoxide but you need to visit directories so that they get entered into autojump’s database before you can jump to them.

j is a wrapper for autojump, jc jumps to subdirectories of the current directory.

You can find the full usage documentation here.

Fuzzy finder: `fzf`

fzf allows to find elements of any list through incremental completion and fuzzy matching. It can be paired with any number of commands.

Installation

fzf is installed on the Alliance clusters.

To get fzf kbds and fuzzy completion in your shell, add to your .bashrc:

eval "$(fzf --bash)"

and/or your .zshrc:

source <(fzf --zsh)

To install it on your machine, you can follow the instructions here.

Direct usage

If you run fzf directly, it will search the current directory recursively, do a narrowing selection, and print the result:

fzf

You can make use of fd (or whatever command you want) to filter out unnecessary entries by setting the FZF_DEFAULT_COMMAND environment variable (in your .bashrc or .zshrc):

export FZF_DEFAULT_COMMAND='fd -u --ignore-file /home/marie/.fdignore'

Running fzf by itself will now be equivalent to running your command of choice that you put in the FZF_DEFAULT_COMMAND environment variable piped to fzf. For all other ways to use fzf (see below), this has no effect so you still have the full flexibility of the tool.

`fzf` key bindings

fzf ships with three default kbds:

Ctl+t ➔ paste the selection into the command
Ctl+r ➔ paste the selection from history into the command
Alt+c ➔ cd into the selection

The one that I find particularly useful is the first one. It is extremely useful to start typing a command (e.g. cp), then use Ctl+t to add the path to a file or directory you want to use as argument of the command before finishing to type your command. Try it out!

Pipe to `fzf`

You can pipe the output of any command that returns a list of elements into fzf to find any particular element easily through incremental completion.

For instance, you can pipe the output of ls into fzf:

ls | fzf

To look for a running process:

alias proc='ps -ef | fzf --cycle -i -e +s --tac --reverse'

proc

To kill a running process:

proc_kill() {
    local pid
    pid=$(ps -ef |
          sed 1d |
          fzf --cycle --reverse -i -e -m --bind "ctrl-o:toggle-all" \
          --header "Tab: toggle, C-o: toggle-all" |
          awk '{print $2}')
    echo $pid | xargs kill -${1:-15}
}

proc_kill

To search your command history:

his() {
    fc -ln 1 |
    rg -v '^q$|^x$|^vs$|^ek .*$|^zoom$|^c$|^cca$|^rs ...$|^hobu$|^cd$|^kzo$|^ih.?$|^zre$|^j m$|^y$|^g$|^-$|^auradd$' |
    fzf --cycle -i -e +s --tac --reverse |
    sed 's/ *[0-9]* *//'
}

his

To search your command history and run the selected entry:

his_run() {
    $(fc -ln 1 |
      rg -v '^q$|^x$|^vs$|^ek .*$|^zoom$|^c$|^cca$|^rs ...$|^hobu$|^cd$|^kzo$|^ih.?$|^zre$|^j m$|^y$|^g$|^-$|^auradd$' |
      fzf --cycle -i -e +s --tac --reverse |
      sed 's/ *[0-9]* *//')
}

his_run

fzf has many options to select the order of entries, the type of completion, whether or not to display a preview, etc. You can find the information in the documentation.

File system TUIs

What is a TUI?

Terminal user interfaces (TUIs) are the predecessors to graphical user interfaces (GUIs) which are entirely text based and run in terminals.

They have remained very popular among command line aficionados because they are fast, efficient, powerful, and keyboard-driven, while being friendly and visual.

Fantastic modern ones keep being built for tasks as diverse as interfaces to Git, music players, games, emails, dashboards, and, for our purpose here, file system management.

Formerly most popular file system TUIs

There are many file system TUIs and all of them are actually really good. The two most notable ones used to be:

ranger

Extremely sophisticated, easy to customize, tons of features.

Built in Python, it can be slow for operations in directories with thousands of files.

Minimalist and very fast (written in C).

Not easy to customize (many customizations require compiling from source). Most functionalities rely on plugins that need to be installed. Not easy to get started with.

The new kid: `yazi`

yazi is a brand new file system TUI that has quickly become the most popular out there.

It is extremely modern, very fast (written in Rust), very well documented, intuitive, easy to customize, and integrates with modern utilities such as fd, rg, zoxide, and fzf out of the box.

Only at version 0.4, it is not fully mature yet, but it has already more stars on GitHub than ranger and nnn because it combines ease of customization and sophistication with speed.

Alternatives

Here are other really good—just not as fast or full-featured—file system TUIs, in decreasing number of stars on GitHub:

broot
superfile
lf
xplr
fff (now archived)
vifm
mc

Webinar

I gave a webinar on this topic that you can find here.

How to choose tools?

ls in colours: eza

Installation

On your own machine

On the Alliance clusters

Usage

Alias

Alternative

A cat with wings: bat

Installation

Usage

Faster find: fd

Installation

Basic usage

Print all files in some directory to stdout

Options

Excluded files and directories

RIP grep: ripgrep

Installation

Usage

Smart cd: zoxide

Installation

Usage

Alternative

Installation

Usage

Fuzzy finder: fzf

Installation

Direct usage

fzf key bindings

Pipe to fzf

File system TUIs

What is a TUI?

Formerly most popular file system TUIs

The new kid: yazi

Alternatives

Webinar

`ls` in colours: `eza`

A `cat` with wings: `bat`

Faster `find`: `fd`

RIP `grep`: ripgrep

Smart `cd`: `zoxide`

Fuzzy finder: `fzf`

`fzf` key bindings

Pipe to `fzf`

The new kid: `yazi`