Page Comparison

...

The Engineering Research Compute Cluster (ENGR Cluster) is a medium-scale installation of heterogeneous compute nodes, supporting both common computational tasks, specific Engineering software applications, and specialized nodes for specific research group tasks.

Excluding faculty/lab owned equipment, the structure of the ENGR cluster is as below, broken down into resources by queue.

Compute nodes are organized into queues into which you submit jobs. Some special queues exist for particular labs or administrative purposes, and are not included here, nor are resources that are in generic queues but locked to specific lab populations.

See The LSF Scheduler for more information on managing jobs.

Accessing the Research Compute Cluster

...

where "piname" is the WUSTL Key login of the PI or storage owner. 

Access is granted via WUSTL Key - on the ENGR cluster, this translates to having a valid Kerberos ticket. If you’ve SSH’d into a ENGR host with your WUSTL Key, you will have a valid ticket. If you have logged in with an SSH Key, you will not.

To generate or refresh a Kerberos ticket, use the command

kinit

and enter your WUSTL Key when prompted.

If you habitually leave live connections to the terminal machines, you may want to get into the habit of “kinit”ing your ticket before submitting a job.

Long Jobs With RIS Storage

If you are using RIS storage with your job, and the ticket expires, this will break the job. In order to avoid this, you can generate a keytab file that allows the cluster to renew your Kerberos ticket for much longer - this keytab will last until you change your WUSTL Key password, at which point it must be regenerated.

VNC Sessions, Jupyter Notebooks, VSCode IDEs and RIS Storage

At the current time, VNC jobs do not come pre-armed with your Kerberos key from your initial login to the web service. If you have created a keytab as described above, it will initialize when the VNC session starts. Otherwise, if you wish to access the RIS storage from inside a VNC session, please do

kinit

You will be prompted for your WUSTL Key. You should also do this before submitting a LSF job from inside a VNC session. After you authenticate, you will have an active Kerberos key and will be able to access the RIS storage mounts.

Open OnDemand

The OpenOnDemand interface is at https://compute.engr.wustl.edu - log in with your WUSTL Key, using Duo 2FA.

Files

OpenOndemand provides a file browser interface. Please note at this time, the file browser cannot access RIS storage.

Within the file brwoser you can upload, download, and move files. You can also edit plain text files within the browser, or open the current location in a web-based terminal.

VNC Desktops

There are several VNC sessions available. Many are marked with a specific PI’s name, and are only accessible to users within that PI’s lab.

Cluster Desktop - Virtual GL

This will start on one of three GPU hosts with older cards, specifically set up to allow you to tie your VNC session to a GPU in order to allow GUI applications to correctly render using the GPU.

When you start your GUI application, prepend the command with ‘virtualgl’ - for example

virtualgl glxgears

Jupyter Notebooks

Custom iPython Kernels

The Jupyter notebooks start with the Anaconda used if you execute ‘module add seas-anaconda3’ in a terminal. From there you can build a custom Anaconda environment.

It’s not recommended to have Anaconda add itself to your .bashrc if you use VNC, as it interferes with the ability for the VNC environment to start.{: style=”color: red; opacity: 0.80;” }

Inside the new Anaconda environment, you can then execute

ipython kernel install --user --name=envname

Start a new Jupyter session, and you can then find that kernel from the ‘New’ dropdown within Jupyter.

Extra Jupyter Arguments

You may place extra arguments for Jupyter's startup in this field. Common options include:

--NotebookApp.notebook_dir=/storage1/piname/Active

The above starts Jupyter in a specific directory. You must have a keytab established, as described above, for this to work on RIS storage locations.

Batch Jobs

Batch jobs are the most efficient way to perform computations on the cluster - you can submit a job script file, which will then run on a compute node that meets your requirements. It runs unassisted, without needing monitoring.

Software on the Compute Cluster

...