Quick-Start

srun

1. Set variables to suppress libcuda.so.1 errors when NOT executing code on GPU nodes

   export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda

2. Load OpenMPI module

   module load ris module load openmpi/5.0.5

3. Download and compile sample code

   1. C

      srun -A compute2-account -p general-short \ wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

   2. Fortran

      srun -A compute2-account -p general-short \ wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifort -o mpi_hello_world_f ./mpi_hello.f

4. Execute sample hello_world binaries across nodes via slurm.

   1. C

      srun -A compute2-account -p general-short -N <num_nodes> ./mpi_hello_world_c
      [gunnar@c2-login-001 ~]$ srun -N 4 ./mpi_hello_world_c srun: job 19414 queued and waiting for resources srun: job 19414 has been allocated resources Hello from task 3 on c2-node-004! Hello from task 2 on c2-node-003! Hello from task 0 on c2-node-001! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-002!

   2. Fortran

      srun -A compute2-account -p general-short -N <num_nodes> ./mpi_hello_world_f
      [gunnar@c2-login-001 ~]$ srun -N 4 ./mpi_hello_world_f srun: job 19410 queued and waiting for resources srun: job 19410 has been allocated resources Hello from task 1 on c2-node-002 Hello from task 2 on c2-node-003 Hello from task 3 on c2-node-004 Hello from task 0 on c2-node-001 MASTER: Number of MPI tasks is: 4

sbatch

1. Create an sbatch file in your home directory to define the job.

   #!/bin/bash #SBATCH -A compute2-account #SBATCH --job-name=openmpi-helloworld #SBATCH --partition=general-short #SBATCH --output=%x-%j.out #SBATCH --nodes=4 #SBATCH --time=5 # Disable CUDA errors on CPU nodes export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda # load modules module load ris module load openmpi/5.0.5 # C Tests echo "Running C test" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c mpicc -o mpi_hello_world_c ./mpi_hello.c srun ./mpi_hello_world_c rm -f mpi_hello.c mpi_hello_world_c # Fortran Tests echo "Running Fortran test" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f mpifort -o mpi_hello_world_f ./mpi_hello.f srun ./mpi_hello_world_f rm -f mpi_hello.f mpi_hello_world_f

2. Execute the sbatch job using sbatch

   sbatch /path/to/sbatch_file
   [gunnar@c2-login-002 ~]$ sbatch openmpi-helloworld.slurm Submitted batch job 149326

3. Monitor job output in the .out file within the current working directory.

   tail -f <job_name>-<job_id>.out
   [gunnar@c2-login-002 ~]$ tail -f openmpi-helloworld-149326.out 2025-09-05 12:41:10 (173 MB/s) - ‘mpi_hello.f’ saved [913/913] 2025-09-05 12:41:10 (90.2 MB/s) - ‘mpi_hello.f’ saved [913/913] Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4 Hello from task 2 on c2-node-063 Hello from task 3 on c2-node-064 Hello from task 1 on c2-node-062

4. There will be a section for the C code and a section for the Fortran code

   [gunnar@c2-login-002 ~]$ grep -i running openmpi-helloworld-149326.out Running C test Running Fortran test

salloc

1. Load relevant modules

   module load ris module load openmpi/5.0.5

2. Set variables to suppress libcuda.so.1 errors when NOT executing code on GPU nodes

   export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda

3. Request the pool of resources be allocated to you

   salloc -A compute2-account --partition=general-short --nodes=4
   [gunnar@c2-login-002 ~]$ salloc --partition=general-short --nodes=4 salloc: Pending job allocation 149327 salloc: job 149327 queued and waiting for resources salloc: job 149327 has been allocated resources salloc: Granted job allocation 149327 salloc: Waiting for resource configuration salloc: Nodes c2-node-[061-064] are ready for job [gunnar@c2-login-002 ~]$

4. Download and compile sample code

   1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

   2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifort -o mpi_hello_world_f ./mpi_hello.f

5. Execute sample hello_world binaries across nodes via slurm.

   1. C

      mpirun ./mpi_hello_world_c
      [gunnar@c2-login-002 ~]$ mpirun ./mpi_hello_world_c Hello from task 1 on c2-node-062! Hello from task 3 on c2-node-064! Hello from task 2 on c2-node-063! Hello from task 0 on c2-node-061! MASTER: Number of MPI tasks is: 4

   2. Fortran

      mpirun ./mpi_hello_world_f
      [gunnar@c2-login-002 ~]$ mpirun ./mpi_hello_world_f Hello from task 2 on c2-node-063 Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4

C2-THPC

Terminal

srun

1. Start a C2-THPC job via srun

   CONTAINER_IMAGE='ghcr.io#washu-it-ris/ris-thpc:rocky9.2' MOUNTS_SYSLMOD='/etc/profile.d,/etc/sysconfig/modules' MOUNTS_THPC_DEFAULT='/opt/thpc,/storage2/fs1,/cm,/lib64/libmunge.so.2,/run/munge' MOUNTS_RIS_STORAGE='/storage2/fs1,/scratch2/fs1,/storage1/fs1,/rdcw/fs1,/rdcw/fs2' srun \ -A compute2-account \ --container-image=$CONTAINER_IMAGE \ --container-mounts=$MOUNTS_SYSLMOD,$MOUNTS_THPC_DEFAULT,$MOUNTS_RIS_STORAGE \ --container-workdir=$HOME \ --cpus-per-task=1 \ --ntasks=4 \ --nodes=1 \ --partition=general-short \ --pty bash

2. Set variables to suppress libcuda.so.1 errors when NOT executing code on GPU nodes

   export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda

3. Load the relevant modules

   module load ris module load openmpi/5.0.5

4. Download and compile sample code

   1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

   2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifort -o mpi_hello_world_f ./mpi_hello.f

5. Execute sample hello_world binaries via mpirun within the Slurm job.

   1. C

      mpirun ./mpi_hello_world_c
      [gunnar@c2-node-066 ~]$ mpirun -np 4 ./mpi_hello_world_c Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

   2. Fortran

      mpirun ./mpi_hello_world_f
      [gunnar@c2-node-066 ~]$ mpirun ./mpi_hello_world_f Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

sbatch

1. Create an sbatch file in your home directory to define the job.

   #!/bin/bash #SBATCH -A compute2-account #SBATCH --job-name=openmpi-helloworld #SBATCH --partition=general-short #SBATCH --output=%x-%j.out #SBATCH --nodes=4 #SBATCH --time=5 #SBATCH --container-image='ghcr.io#washu-it-ris/ris-thpc:rocky9.2' #SBATCH --container-mounts='/etc/profile.d,/etc/sysconfig/modules,/opt/thpc,/storage2/fs1,/cm,/lib64/libmunge.so.2,/run/munge,/storage2/fs1,/scratch2/fs1,/storage1/fs1,/rdcw/fs1,/rdcw/fs2' #SBATCH --container-workdir=$HOME # Disable CUDA errors on CPU nodes export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda # load modules module load ris module load openmpi/5.0.5 # C Tests echo "Running C test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c mpicc -o mpi_hello_world_c ./mpi_hello.c srun ./mpi_hello_world_c rm -f ./mpi_hello.c ./mpi_hello_world_c # Fortran Tests echo "Running Fortran test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f mpifort -o mpi_hello_world_f ./mpi_hello.f srun ./mpi_hello_world_f rm -f mpi_hello.f mpi_hello_world_f

2. Execute the sbatch job using sbatch

   sbatch /path/to/sbatch_file
   [gunnar@c2-login-002 ~]$ sbatch openmpi-helloworld.slurm Submitted batch job 150621

3. Monitor job output in the .out file within the current working directory.

   tail -f <job_name>-<job_id>.out
   [gunnar@c2-login-002 ~]$ tail -f openmpi-helloworld-150621.out srun: Step created for StepId=150621.1 pyxis: imported docker image: ghcr.io#washu-it-ris/ris-thpc:rocky9.2 pyxis: imported docker image: ghcr.io#washu-it-ris/ris-thpc:rocky9.2 pyxis: imported docker image: ghcr.io#washu-it-ris/ris-thpc:rocky9.2 pyxis: imported docker image: ghcr.io#washu-it-ris/ris-thpc:rocky9.2 Hello from task 0 on c2-node-069 MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-070 Hello from task 3 on c2-node-072 Hello from task 2 on c2-node-071

4. There will be a section for the C code and a section for the Fortran code

   [gunnar@c2-login-002 ~]$ grep -i running openmpi-helloworld-150621.out Running C test from: /home/gunnar Running Fortran test from: /home/gunnar

Open OnDemand

1. Start a C2-THPC job in OOD

2. Set variables to suppress libcuda.so.1 errors when NOT executing code on GPU nodes

   export OMPI_MCA_accelerator=^cuda export OMPI_MCA_rcache=^gpusm,rgpusm export OMPI_MCA_btl=^smcuda

3. Load the relevant modules

   module load ris module load openmpi/5.0.5

4. Download and compile sample code

   1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

   2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifort -o mpi_hello_world_f ./mpi_hello.f

5. Execute sample hello_world binaries via mpirun within the Slurm job.

   1. C

      mpirun -np <num_cpus> ./mpi_hello_world_c
      [gunnar@c2-node-066 ~]$ mpirun -np 4 ./mpi_hello_world_c Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

   2. Fortran

      mpirun -np <num_cpus> ./mpi_hello_world_f
      [gunnar@c2-node-066 ~]$ mpirun ./mpi_hello_world_f Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

Versions

5.0

Currently this is the only version of OpenMPI installed into the x86_64 Spack environment and directly supported by RIS AppEng.