Modern utilities

In recent years, a number of open-source utilities for the Unix shell have emerged. Some are meant as replacements for classic tools with improved performance, better defaults, or nicer-looking outputs; others add novel functionality. Several of them were recently installed on the Alliance clusters.

In this section we will cover a selection of tools that are very popular, well-maintained, and will improve your time on the command line.

ls in colours: eza

eza is a replacement for ls.

For the most part, it adds colours and uses the same options as ls, but it also adds a few features and has better default options.

Installation

On your own machine

You can find the installation instructions here.

On the Alliance clusters

eza is not installed on the Alliance clusters, so you will have to install it locally under your own user. This is easy to do because it is written in Rust and can be installed with the Rust package manager.

First, you load a Rust module, then you install the package:

module load rust/1.76.0
cargo install eza

Then, you have to make sure that ~/.cargo/bin is in your PATH as this is where the binary will get installed.

Add to your .bashrc:

export PATH=$PATH:~/.cargo/bin

You only need to do this once in the production cluster you use. After eza has been installed, it will be accessible on subsequent sessions.

Usage

eza

projects scratch

You now have different colours for different types of files. You can already see that the directory (projects) is in bold blue and the symlink (scratch) is in cyan (my website doesn’t render these colours but you will see them when you run the command in our training cluster).

Let’s create some files quickly to see a bit more colours:

touch test.html test.md test.py test.R
eza

projects scratch test.html test.md test.py test.R

Now you can see that the scripts are in bold yellow while the markup files are in gray.

The flags are similar to those of ls, but some defaults are slightly different:

eza -al

.rw-------@ 2.8k user02 19 Dec 04:20 .bash_history
.rw-------@   18 user02 13 Dec 00:55 .bash_logout
.rw-r-----@  141 user02 19 Dec 04:20 .bash_profile
.rw-r-----@ 7.3k user02 19 Dec 03:57 .bashrc
drwxr-x---@    - user02 19 Dec 03:57 .cargo
drwx------@    - user02 13 Dec 18:10 .ssh
drwxr-xr-x@    - root   13 Dec 00:57 projects
lrwxrwxrwx@    - user02 13 Dec 00:55 scratch -> /scratch/user02
.rw-r-----@    0 user02 19 Dec 04:25 test.html
.rw-r-----@    0 user02 19 Dec 04:25 test.md
.rw-r-----@    0 user02 19 Dec 04:25 test.py
.rw-r-----@    0 user02 19 Dec 04:29 test.R

Compare with ls -al:

total 60
drwx------.  12 user02 user02 4096 Dec 19 04:29 .
drwxr-xr-x. 101 root   root   4096 Dec 13 00:57 ..
-rw-------.   1 user02 user02 2763 Dec 19 04:20 .bash_history
-rw-------.   1 user02 user02   18 Dec 13 00:55 .bash_logout
-rw-r-----.   1 user02 user02  141 Dec 19 04:20 .bash_profile
-rw-r-----.   1 user02 user02 7320 Dec 19 03:57 .bashrc
drwxr-x---.   4 user02 user02  125 Dec 19 03:57 .cargo
drwx------.   2 user02 user02   48 Dec 13 18:10 .ssh
drwxr-xr-x.   2 root   user02   27 Dec 13 00:57 projects
lrwxrwxrwx.   1 user02 user02   15 Dec 13 00:55 scratch -> /scratch/user02
-rw-r-----.   1 user02 user02    0 Dec 19 04:29 test.R
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.html
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.md
-rw-r-----.   1 user02 user02    0 Dec 19 04:25 test.py

eza by default shows the output in a human readable format (-h flag with ls) and without the group (-G with ls).

There is also the additional -T flag which is equivalent to running the tree utility:

eza -T

.
├── projects
│   └── def-sponsor00 -> /project/def-sponsor00
├── scratch -> /scratch/user02
├── test.html
├── test.md
├── test.py
└── test.R

Alias

If you want, you can alias it to ls by adding to your .bashrc file:

alias ls=eza

If you ever want to use the true ls utility, you can do so with \ls.

If you want a simpler and more lightweight way to add colours to your ls outputs, you can look at LS_COLORS. I did this for years until I found eza.

To install it locally in the Alliance clusters, you have to download and uncompress a script, and copy it to a proper location:

mkdir ./LS_COLORS &&
    curl -L https://api.github.com/repos/trapd00r/LS_COLORS/tarball/master | tar xzf - --directory=./LS_COLORS --strip=1 &&
    mkdir -p ~/.local/share &&
    cp ~/LS_COLORS/lscolors.sh ~/.local/share &&
    rm -r ~/LS_COLORS

Then you need to add the following to your .bashrc file to source the script and add a proper alias to ls:

source ~/.local/share/lscolors.sh
alias ls='ls --color'

A cat with wings: bat

bat is a replacement for cat with syntax highlighting and line number.

Installation

rg is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

bat .bash_profile

───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .bash_profile
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ # .bash_profile
   2   │
   3   │ # Get the aliases and functions
   4   │ if [ -f ~/.bashrc ]; then
   5   │     . ~/.bashrc
   6   │ fi
   7   │
   8   │ # User specific environment and startup programs
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────

RIP grep: ripgrep

ripgrep provides the rg utility—is a replacement for grep with greatly improved performance.

Installation

rg is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

Search lines in a file containing a pattern

rg .bash .bash_profile

1:# .bash_profile
4:if [ -f ~/.bashrc ]; then
5:  . ~/.bashrc

You can find full details on the syntax here.

Faster find: fd

fd is a replacement for find with improved performance is simpler syntax.

Installation

fg is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

Search file names in current directory recursively for a pattern:

fd te

test.R
test.html
test.md
test.py

Search file names in another directory recursively for a pattern:

fd te projects/

Search for files with a particular file extension:

fd -e md

test.md

You can find full details on the syntax here.

Smart cd: zoxide

zoxide allows to easily jump to any directory.

Installation

Instructions to install zoxide on your machine can be found here. Note that fzf (see below) adds cool functionality to it, so you might want to install it as well.

Installing zoxide on the Alliance clusters is extremely easy.

1. First, you have to run the command:

curl -sSfL https://raw.githubusercontent.com/ajeetdsouza/zoxide/main/install.sh | sh

This will install the binary in ~/.local/bin.

2. Then, you have to make sure that ~/.local/bin is in your PATH.

Add to your .bashrc:

export PATH=$PATH:~/.local/bin

3. Finally, you need to add to your .bashrc file (if you want to use it in Bash):

eval "$(zoxide init bash)"

Or, if you want to use it in Zsh, add to your .zshrc file:

eval "$(zoxide init zsh)"

You can change the z and zi commands (a bit hard to type) by whatever command you want. I personally changed them to j and ji (“j” for “jump”). To do this, replace the eval expression(s) above by:

eval "$(zoxide init --cmd j bash)"

and/or

eval "$(zoxide init --cmd j zsh)"

Usage

Now, you can type z (or whatever command you chose—see above) instead of a regular cd command.

In addition, you can simplify the path you want to get to to just a few characters from its path and zoxide will take you there.

If there are multiple possible location matching your entry, the matching algorithm will chose the highest ranking path based on your frequency of use and how recently you visited a path.

This means that you can visit your usual places extremely easily. For less frequent places, add a bit more info.

Finally, if you want to choose amongst all possible options in a completion framework, use zi instead and zoxide will open fzf (see below).

A tool that served me well until someone pointed the faster and better zoxide to me is autojump.

Installation

autojump is installed on the Alliance clusters, but you need to source a script before you can use it. Since you need to do this at every session, you should add to your .bashrc file:

[[ -s $EPREFIX/etc/profile.d/autojump.sh ]] && source $EPREFIX/etc/profile.d/autojump.sh

To install autojump on your machine, you can follow the instructions here.

Usage

You first need to visit directories so that they get entered in a database kept by autojump. This database also keeps track of how often each directory gets visited. This allows you to jump to the directories you visit all the time by simply typing a few characters of their name.

j is a wrapper for autojump.

jc can be used to jump to subdirectories of the current directory.

Since we have so few directories in our users’ accounts on the training cluster and we haven’t built the database yet by navigating our respective files systems, I will demo this tool on my machine.

You can find the full usage here.

Fuzzy finder: fzf

fzf is a fuzzy finder that can be paired with any number of commands.

Installation

fzf is installed on the Alliance clusters. To install it on your machine, you can follow the instructions here.

Usage

You can pipe the output of any command that returns a list of elements into fzf to find any particular element easily through incremental completion.

For instance, you can pipe the output of ls into fzf:

ls | fzf

To look for a running process:

alias proc='ps -ef | fzf --cycle -i -e +s --tac --reverse'

proc

To kill a running process:

proc_kill() {
    local pid
    pid=$(ps -ef |
          sed 1d |
          fzf --cycle --reverse -i -e -m --bind "ctrl-o:toggle-all" \
          --header "Tab: toggle, C-o: toggle-all" |
          awk '{print $2}')
    echo $pid | xargs kill -${1:-15}
}

proc_kill

To search your command history:

his() {
    fc -ln 1 |
    grep -Ev '^q$|^x$|^vs$|^ek .*$|^zoom$|^c$|^cca$|^rs ...$|^hobu$|^cd$' |
    fzf --cycle -i -e +s --tac --reverse |
    sed 's/ *[0-9]* *//'
}

his

To search your command history and run the selected entry:

his_run() {
    $(fc -ln 1 |
      grep -Ev '^q$|^x$|^vs$|^ek .*$|^zoom$|^c$|^cca$|^rs ...$|^hobu$|^cd$' |
      fzf --cycle -i -e +s --tac --reverse |
      sed 's/ *[0-9]* *//')
}

his_run

fzf has many options to select the order of entries, the type of completion, whether or not to display a preview, etc. You can find the information in the documentation.

Webinar

I will give a webinar on these tools on January 28.