RIS Compute 102

In a normal HPC environment, batch scripts tell the system how to process your data.

We will cover shell scripts first and then batch scripts as batch scripts are shell scripts with additional functionality.

This type of setup is typical for the average HPC.

In the RIS HPC environment, you do not need batch scripts, but you can essentially duplicate them for the sake of familiarity.

These are a text file with a series of Unix/Linux commands.

They are useful for automating a series of commands.

Each shell has its own syntax but they all support programming structures like loops, conditionals, etc.

They are powerful enough that it becomes possible to create what are essentially new commands.

Example of a simple shell script:

Text editors are used to write and edit shell/batch scripts (as well as other files).

There are many types of text editors - some are CLI and some are GUI.

A few text editors vi, vim, emacs, gedit, etc.

Text editors that are CLI based are tricky to learn, so a GUI text editor can be useful for the beginning user.

If you edit any file on Windows and transfer it to the HPC environment, you will have formatting errors.

Windows terminates each line with both a line feed and a carriage return while Unix/Linux just uses a line feed.

X11 is part of the Unix/Linux system and allows for GUIs.

This not only allows for you to run GUIs directly on a Unix system, you can also run GUIs on remote systems and have them display on your local system.

Linux has X11 built into the OS.

Unix has X11 included in OSX 10.5 - 10.7, otherwise you will need to install XQuartz (https://www.xquartz.org/).

Windows, you can use MobaXterm to utilize X11.

SSH can “tunnel” X11 so that it travels through a secure connection.

You just have to add the -Y flag to your connection command when using Linux/Unix.

MobaXterm has an SSH client that will automatically tunnel X11 so we don’t need the -Y flag when using that particular software to connect.

Emacs is a good basic editor. It looks like notepad and has many basic features.

Write the script in a text editor

Make the script executable

Run the script

Lines that begin with ‘#’ are usually interpreted as comments.

An important exception to this is the first line of a shell script.

'#!' tells the OS what interpreter to run the shell script with, there are multiple.

• Bourne:

  #!/bin/sh

• BASH:

  #!/bin/bash

• Python:

  #!/usr/bin/python

• Perl:

  #!/usr/bin/perl -w

To execute a Unix/Linux command from a shell script, you just enter the command on a new line in your script.

echo is used to print text to the screen.

Example of writing a simple shell script and executing said script:

You can use the chmod command to make your script executable.

In the command line, cat and less are commands that can be used to view the contents of a file. cat will just print the entire contents of the file to the screen while less will allow you to interact with the file and scroll through the contents.

When you are viewing the file with less, you can use the spacebar to scroll through the content and ‘q’ to quit out of the view.

As mentioned in the RIS Compute 101, the command ls -lh returns the details of all files and directories in the current directory (working directory).

The first section of information contains whether or not the item listed is a file or a directory (- = file, d = directory).

It also contains the permissions for 3 groups, the first is the owner of the file, the second is the group the owner of the file is in, and the third is for everyone.

The permission codes are as follows:

r (read) / w (write) / x (executable)

The following would represent that only the owner can read, write, and execute the file.

The following represents the owner has permissions as above, but the group also has read and write permissions.

The following represents as above but now everyone can read as well.

We’ve shown the chmod command before which allowed us to make our shell script executable, the same command can be used to change any of the permissions of a file or directory.

There is a number system associated with the permissions.

Read is 4, Write is 2, and Executable is 1, therefore 4 would mean only read, 6 would mean read and write and 7 would mean all three.

The following would change the permissions so that only the owner can read and write.

The following would make it so that the owner and the group can read and write.

The following would make it so that everyone can read the file and only the owner and group can write.

To run a script you need to enter the path to the script. You can use either the absolute path, or the home alias.

And if your script is in the directory you’re currently in (working directory) you can use “./” as the “.” indicates the current directory (working directory). This is what we used in our example.

In this section, we’ll briefly discuss variables, conditionals, and loops.

In shell scripts, traditionally variables are defined using all capital letters.

Putting a dollar sign in front of a variable indicates to give the value of the variable.

You can use variables within scripts. With this you can do things like only type a long path to a file once and then reference the variable later in the script instead of having to type the long path again.

Conditionals are where different actions are taken dependent on whether a certain condition is met or not.

An example of this are if/then statements.

Loops execute a command over and over again a specified amount of times.

If you have a defined set of values, you would use a for loop.

If you need to loop until a specified condition is no longer valid, you would use a while loop.

The following while loop runs while the counter is less than 10 and simply echos the value of the counter.

The following for loops iterates on i with values 1 through 5 and echos Welcome $i times for each value in the list (1,2,3,4,5).