Our workflow for this course

Author

Marie-Hélène Burle

We are providing a virtual training cluster for this course with the necessary modules already loaded and the packages already installed. That virtual training cluster mimics the Alliance clusters (but with very little resources) so that you can practice workflows that you will later use on them.

In some parts of the course, we will use JupyterHub, in other parts, we will log in via SSH.

This section describes both. You can refer to it when needed during the course.

Get the info

During the course, we will give you 4 pieces of information:

  • a link to a list of usernames,
  • the URL of a JupyterHub for this course,
  • the hostname for the temporary training cluster for this course,
  • the password to access that cluster.

You need to claim a username by adding your first name or a pseudo next to a free username on the list to claim it.

Your username is the name that was already on the list, NOT what you wrote next to it (which doesn’t matter at all and only serves at signalling that this username is now taken).

Your username will look like userxxx—xxx being 3 digits—with no space and no capital letter.

JupyterHub

Log in

Open the JupyterHub URL we gave you in your browser, then use the username you claimed and the password we gave you to log in.

Choose server options

In the Server Options page that opens, select what we tell you for various parts of the course. Please do not request more resources as this would prevent others to get access (remember that this virtual cluster runs on limited resources just for this course).

If you would like to make a change to the information you entered on the server option page after you have pressed start, log out (click on File in the top menu and select Log out at the very bottom), log back in, edit the server options, and press start again.

Start a Python notebook

To start a Jupyter notebook with the Python kernel, click on the button Python 3 in the Notebook section (top row of buttons).

The Jupyter interface

In a fashion Vi users will be familiar with, Jupyter notebooks come with two modes: edit mode in which you can type text as usual and command mode in which many keys are shortcuts to specific actions.

Here are some useful key bindings to navigate a Jupyter notebook:

Enter            enter edit mode
Esc              enter command mode

# in edit mode
Tab              code completion

# in command mode
up               navigate up
down             navigate up
Shift+up         select multiple cells up
Shift+down       select multiple cells down
a                insert a new blank cell above
b                insert a new blank cell below
c                copy the current or selected cells
x                cut the current or selected cells
v                paste the copied or cut cells
m                turn the cell into a markdown cell
y                turn the cell into a code cell
Shift+m          merge selected cells

# in either mode
Ctl+Enter        run the current cell
Shift+Enter      run the current cell and move to a new cell below

Log out

When you are done with a session in the JupyterHub, please log out. This will release the resources and make them available to use via SSH.

To log out, click on File in the top menu and select Log out at the very bottom.

SSH

Why use SSH?

Because GPUs are in high demand, you need to learn how to use them optimally without letting them sit idle.

The key is to write scripts and submit jobs to Slurm instead of using an interactive session such as Jupyter: when you use Jupyter, the resources are made available to you during the whole session, but you are actually running them only when you execute code cells. While you are thinking, typing code, etc. they are poorly used.

So while Jupyter is fantastic to prototype code on small data, few samples, etc. using little resources, once your code is ready and you want to run it at scale, you have to submit batch jobs to the Slurm scheduler.

Log in

Run the ssh command

Linux users:   open the terminal emulator of your choice.
macOS users:   open “Terminal”.

Then type:

ssh userxxx@hostname

and press Enter.

  • Replace userxxx by your username (e.g. user009).
  • Replace hostname by the hostname we will give you the day of the workshop.

When asked:

Are you sure you want to continue connecting (yes/no/[fingerprint])?

Answer: “yes”.

We suggest using the free version of MobaXterm, a software that comes with a terminal emulator and a GUI interface for SSH sessions.

Here is how to install MobaXterm:

  • download the “Installer edition” to your computer (green button to the right),
  • unzip the file,
  • double-click on the .msi file to launch the installation.

Here is how to log in with MobaXterm:

  • open MobaXterm,
  • click on Session (top left corner),
  • click on SSH (top left corner),
  • fill in the Remote host * box with the cluster hostname we gave you,
  • tick the box Specify username,
  • fill in the box with the username you selected (e.g. user009),
  • press OK,
  • when asked Are you sure you want to continue connecting (yes/no/[fingerprint])?, answer: “yes”.

Here is a live demo.

Enter the password

When prompted, enter the password we gave you.

You will not see anything happen as you type the password. This is normal and it is working, so keep on typing the password.

This is called blind typing and is a Linux safety feature. It can be unsettling at first not to get any feed-back while typing as it really looks like it is not working. Type slowly and make sure not to make typos.

Then press Enter.

Am I logged in?

To know whether or not you are logged in, look at your prompt: it should look like the following (with your actual username):

[userxxx@login1 ~]$

Troubleshooting

Problems logging in are almost always due to typos. If you cannot log in, retry slowly, entering your password carefully.

How do I log out?

You can log out by pressing Ctl+d.

For this course, everything is already setup for you. So once you are logged in, you don’t need to install packages, activate virtual environments, or load modules.

Create a directory

Let’s create a directory for this course and enter it:

mkdir cupy
cd cupy

Writing scripts

Use nano or the text editor of your choice to write both the Python scripts that will contain the code to run and the bash scripts that we will submit to Slurm to request resources.

Example:

nano my_script.py