General Guidelines and Introduction:

Please note ‘Login Nodes’ are not compute nodes, they are intended to only for accessing cluster and doing house keeping tasks such as editing files, moving data etc. Usage on login nodes is restricted to 6GB per user per login node. Please read Compute2 guidelines and policies carefully that describes usage policy and best practices.

For doing computation you will have to submit batch jobs or interactive jobs. Or use our Open-OnDemand portal.

Connect via SSH:

You can connect via ssh to any of our three login nodes:
c2-login-001, c2-login-002 and c2-login-003

by running the below command:

where N can be 1,2 or 3.

An example command would be:

Using Slurm

Load RIS module via ml

• See what is available via ml with avail command

• Load RIS module

Job Options

• The help option provides a full list of what's available

• The basic options are listed here

  • Account: required as there is no longer a default set for users.

    • A user can find their accounts by following the information in our Slurm Accounts documentation.

      -A, --account=name charge job to specified account

  • CPU: Use --ntasks for increase of CPUs used with --cpus-per-task=1

  -c, --cpus-per-task=ncpus number of cpus required per task -n, --ntasks=ntasks number of tasks to run

  • Job name

  -J, --job-name=jobname name of job

  • Ram/Memory

  --mem=MB minimum amount of real memory --mem-per-cpu=MB maximum amount of real memory per allocated cpu required by the job. --mem >= --mem-per-cpu if --mem is specified.

  • Host/Node/Server

  -N, --nodes=N number of nodes on which to run (N = min[-max])

  • Partition/Queue

  -p, --partition=partition partition requested

  • Standard Out (stdout)

  -o, --output=out location of stdout redirection

  • Job Run Time

  -t, --time=minutes time limit

  • Using a container (Docker image)

  --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH [pyxis] the image to use for the container filesystem. Can be either a docker image given as an enroot URI, or a path to a squashfs file on the remote host filesystem.

  • Using a GPU

  -G, --gpus=n count of GPUs required for the job --gpus-per-node=n number of GPUs required per allocated node --mem-per-gpu=n real memory required per allocated GPU --cpus-per-gpu=n number of CPUs required per allocated GPU

Using srun

• Submits a job that runs in real time

• Example using python

• Storage allocations are mounted and you do not need to mount storage

  • Storage allocations do need to be mounted if using a container. See the Using Containers section.

• You can get a shell connection to a job by using the --pty option and /bin/bash as the command.

• This works for jobs that use installed applications or containers.

Using sbatch

• Submits a batch job (Runs in the background)

• Create a job file: testjob.slurm

• Submit job using sbatch

• Check the output

• Storage allocations are mounted and you do not need to mount storage.

  • Storage allocations do need to be mounted if using a container. See the Using Containers section.

Using squeue

• Displays running jobs

• Basic options listed here

  • By user

  -u, --user=user_name(s) comma separated list of users to view

  • By partition/queue

  -p, --partition=partition(s) comma separated list of partitions to view, default is all partitions

  • By name

  -n, --name=job_name(s) comma separated list of job names to view

  • By account

  -A, --account=account(s) comma separated list of accounts to view, default is all accounts

• By default shows all jobs

• Use the -u option to show only your jobs

Using scancel

• Cancels or kills running jobs

• Default use job ID to cancel a job

• Use the --me option to kill all your jobs

• Can cancel via other options

  • Job name

  -n, --name=job_name act only on jobs with this name

  • User

  -u, --user=user_name act only on jobs of this user

  • Partion/Queue

  -p, --partition=partition act only on jobs in this partition

Using sinfo

• Provides partitions/queues available

Using GPUs

• GPUs are available in the general and general-short partitions

• Use a script that uses pytorch to test for GPU usage called test_gpu.py

• Load slurm and py-torch module. Shown below is an example command.

• Run a test using a python script

Using Containers

• A container (Docker image) can be used with the following option

• ${Home} directories are automatically mounted

• The ${Home} directory is not default working directory

• Use the full path name for files

• The working directory can be set with the following option

• Example setting the working directory

• Storage allocations are not mounted by default

• Storage allocations can be mounted with the following option.

• Example mounting a storage allocaiton

• These same options can be used with sbatch submissions

Using Job Arrays

• Job arrays can only be used with sbatch jobs

• They can be used with the following option

• An example of using a job arrays

  • Job script

  #!/bin/bash #SBATCH --job-name=array-test #SBATCH --array=1-5 #SBATCH --output=testarray.log #SBATCH -p general-cpu #SBATCH -A compute2-account /bin/bash -c "date;sleep 300;date"

  • Job submission

  $ sbatch testarrayjob.slurm Submitted batch job 243

  • Looking at job arrays in the queue

  $ sbatch testarrayjob.slurm Submitted batch job 243 [elyn@c2-login-001 ~]$ squeue -u elyn JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 243_2 general array-te elyn R 0:09 1 c2-node-001 243_3 general array-te elyn R 0:09 1 c2-node-001 243_4 general array-te elyn R 0:09 1 c2-node-001 243_5 general array-te elyn R 0:09 1 c2-node-001 243_1 general array-te elyn R 0:10 1 c2-node-001

• Job arrays can all be cancelled

  • Example for job array (job ID) 50 with 5 elements

  scancel 50

• Or particular array elements can be cancelled

  scancel 50_1 scancel 50_[1-3] scancel 50_1 50_4