Redirections & pipes

Author

Marie-Hélène Burle

By default, commands that produce an output print to the terminal. This output can however be redirected to be printed elsewhere (e.g. to files) or to be passed as the argument of another command.

This section will cover the most basic cases.

Data for this section (same data as previous section)

For this section, we will play with files created by The Carpentries.

You can download them into a zip file called bash.zip with:

wget http://bit.ly/bashfile -O bash.zip

You can then unzip that file with:

unzip bash.zip

You should now have a data-shell directory with a molecules subdirectory.

cd into it:

cd data-shell/molecules

Redirections

By default, commands that produce an output print it to standard output—that is, the terminal. That is what we have been doing so far.

This output can however be redirected with the > sign. For instance, it can be redirected to a file, which is very handy if you want to save the result.

Example:

Let’s print the number of lines in each .pdb file in the molecules directory:

wc -l *.pdb

  20 cubane.pdb
  12 ethane.pdb
   9 methane.pdb
  30 octane.pdb
  21 pentane.pdb
  15 propane.pdb
 107 total

Your turn:

What does the wc command do?
What does the -l flag for this command do?
How did you find out?

To save this result into a file called lengths.txt, we run:

wc -l *.pdb > lengths.txt

Your turn:

How can you make sure that you did create a file called lengths.txt?

Let’s print its content to the terminal:

cat lengths.txt

  20 cubane.pdb
  12 ethane.pdb
   9 methane.pdb
  30 octane.pdb
  21 pentane.pdb
  15 propane.pdb
 107 total

As you can see, it contains the output of the command wc -l *.pdb.

Of course, we can print the content of the file with modification. For instance, we can sort it:

sort -n lengths.txt

   9 methane.pdb
  12 ethane.pdb
  15 propane.pdb
  20 cubane.pdb
  21 pentane.pdb
  30 octane.pdb
 107 total

And we can redirect this new output to a new file:

sort -n lengths.txt > sorted.txt

Instead of printing an entire file to the terminal, you can print only part of it.

Let’s print the first line of the new file sorted.txt:

head -1 sorted.txt

   9 methane.pdb

Pipes

Another form of redirection is the Bash pipe. Instead of redirecting the output to a different stream for printing, the output is passed as an argument to another command. This is very convenient because it allows to chain multiple commands without having to create files or variables to save intermediate results.

For instance, we could run the three commands we ran previously at once, without the creation of the two intermediate files:

wc -l *.pdb | sort -n | head -1

   9 methane.pdb

In each case, the output of the command on the left-hand side (LHS) is passed as the input of the command on the right-hand side (RHS).

Your turn:

In a directory we want to find the 3 files that have the least number of lines. Which command would work for this?

wc -l * > sort -n > head -3
wc -l * | sort -n | head 1-3
wc -l * | sort -n | head -3
wc -l * | head -3 | sort -n

Here is a video of a previous version of this workshop.