/
C2 Intel OneAPI

You can use our service desk portal for getting RIS support. RIS also offers 15 min. virtual office hours session Mon-Thru..

C2 Intel OneAPI

Quick-Start

srun

  1. Load Intel OneAPI modules for both compilers and mpi

    module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0

  2. Set I_MPI_PMI_LIBRARY to a valid PMI library

    export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so

Intel OneAPI does not have support for PMIX at this time.
PMI2 is preferred over PMI1 for better scaling performance.

  1. Download and compile sample code

    1. C

      srun -A compute2-account -p general-short \ wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

    2. Fortran

      srun -A compute2-account -p general-short \ wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifc -o mpi_hello_world_f ./mpi_hello.f

When using srun to execute code mpirun is not required.

  1. Execute sample hello_world binaries across nodes via Slurm.

    1. C

      srun -A compute2-account -p general-short --mpi=pmi2 -N <num_nodes> ./mpi_hello_world_c
      [gunnar@c2-login-001 ~]$ srun --mpi=pmi2 -N 4 ./mpi_hello_world_c srun: job 19414 queued and waiting for resources srun: job 19414 has been allocated resources Hello from task 3 on c2-node-004! Hello from task 2 on c2-node-003! Hello from task 0 on c2-node-001! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-002!

    2. Fortran

      srun -A compute2-account -p general-short --mpi=pmi2 -N <num_nodes> ./mpi_hello_world_f
      [gunnar@c2-login-001 ~]$ srun --mpi=pmi2 -N 4 ./mpi_hello_world_f srun: job 19410 queued and waiting for resources srun: job 19410 has been allocated resources Hello from task 1 on c2-node-002 Hello from task 2 on c2-node-003 Hello from task 3 on c2-node-004 Hello from task 0 on c2-node-001 MASTER: Number of MPI tasks is: 4

sbatch

  1. Create an sbatch file in your home directory to define the job.

    #!/bin/bash #SBATCH -A compute2-account #SBATCH --job-name=intel-oneapi-helloworld #SBATCH --partition=general-short #SBATCH --output=%x-%j.out #SBATCH --nodes=4 #SBATCH --time=5 # load modules module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0 # export env vars export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so # C Tests echo "Running C test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c mpicc -o mpi_hello_world_c ./mpi_hello.c srun --mpi=pmi2 ./mpi_hello_world_c rm -f mpi_hello.c mpi_hello_world_c # Fortran Tests echo "Running Fortran test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f mpifc -o mpi_hello_world_f ./mpi_hello.f srun --mpi=pmi2 ./mpi_hello_world_f rm -f mpi_hello.f mpi_hello_world_f

  1. Execute the sbatch job using sbatch

    sbatch /path/to/sbatch_file
    [gunnar@c2-login-002 ~]$ sbatch ./intel-oneapi-helloworld.slurm Submitted batch job 149330

  2. Monitor job output in the .out file within the current working directory.

    tail -f <job_name>-<job_id>.out
    [gunnar@c2-login-002 ~]$ tail -f ./intel-oneapi-helloworld-149330.out 2025-09-05 13:02:57 (181 MB/s) - ‘mpi_hello.f’ saved [913/913] 2025-09-05 13:02:57 (193 MB/s) - ‘mpi_hello.f’ saved [913/913] Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4

  3. There will be a section for the C code and a section for the Fortran code

    [gunnar@c2-login-002 ~]$ grep -i -E 'running|hello' ./intel-oneapi-helloworld-149330.out Running C test from: /home/gunnar Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4 Running Fortran test from: /home/gunnar Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4

salloc

  1. Load relevant modules

    module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0

  2. Set I_MPI_PMI_LIBRARY to a valid PMI library

    export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so

  3. Request the pool of resources be allocated to you

    salloc -A compute2-account --partition=general-short --nodes=4
    [gunnar@c2-login-002 ~]$ salloc --partition=general-short --nodes=4 salloc: Pending job allocation 149350 salloc: job 149327 queued and waiting for resources salloc: job 149327 has been allocated resources salloc: Granted job allocation 149350 salloc: Waiting for resource configuration salloc: Nodes c2-node-[061-064] are ready for job [gunnar@c2-login-002 openmpi]$

  4. Download and compile sample code

    1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

    2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifc -o mpi_hello_world_f ./mpi_hello.f

  5. Execute sample hello_world binaries across nodes via Slurm.

    1. C

      mpirun ./mpi_hello_world_c
      [gunnar@c2-login-002 ~]$ mpirun ./mpi_hello_world_c Hello from task 0 on c2-node-061! MASTER: Number of MPI tasks is: 4 Hello from task 3 on c2-node-064! Hello from task 2 on c2-node-063! Hello from task 1 on c2-node-062!

    2. Fortran

      mpirun ./mpi_hello_world_f
      [gunnar@c2-login-002 ~]$ mpirun ./mpi_hello_world_f Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 1 on c2-node-062

C2-THPC

Terminal

srun

Currently limited to only single-node jobs.
Multi-node jobs may function but have not been fully validated and documented.

  1. Start a C2-THPC job via srun

    CONTAINER_IMAGE='ghcr.io#washu-it-ris/ris-thpc:rocky9.2' MOUNTS_SYSLMOD='/etc/profile.d,/etc/sysconfig/modules' MOUNTS_THPC_DEFAULT='/opt/thpc,/storage2/fs1,/cm,/lib64/libmunge.so.2,/run/munge' MOUNTS_RIS_STORAGE='/storage2/fs1,/scratch2/fs1,/storage1/fs1,/rdcw/fs1,/rdcw/fs2' srun \ -A compute2-account \ --container-image=$CONTAINER_IMAGE \ --container-mounts=$MOUNTS_SYSLMOD,$MOUNTS_THPC_DEFAULT,$MOUNTS_RIS_STORAGE \ --container-workdir=$HOME \ --ntasks=4 \ --nodes=1 \ --partition=general-short \ --pty bash

  1. Load the relevant modules

    module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0

  2. Set I_MPI_PMI_LIBRARY to a valid PMI library

    export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so

Intel OneAPI does not have support for PMIX at this time.
PMI2 is preferred over PMI1 for better scaling performance.

  1. Download and compile sample code

    1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

    2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifc -o mpi_hello_world_f ./mpi_hello.f

  1. Execute sample hello_world binaries via mpirun within the Slurm job.

    1. C

      mpirun ./mpi_hello_world_c
      [gunnar@c2-node-066 ~]$ mpirun ./mpi_hello_world_c Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

    2. Fortran

      mpirun ./mpi_hello_world_f
      [gunnar@c2-node-066 ~]$ mpirun ./mpi_hello_world_f Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

sbatch - (WORK IN PROGRESS)

  1. Create an sbatch file in your home directory to define the job.

    #!/bin/bash #SBATCH -A compute2-account #SBATCH --job-name=intel-oneapi-helloworld #SBATCH --partition=general-short #SBATCH --output=%x-%j.out #SBATCH --nodes=4 #SBATCH --time=5 #SBATCH --container-image='ghcr.io#washu-it-ris/ris-thpc:rocky9.2' #SBATCH --container-mounts='/etc/profile.d,/etc/sysconfig/modules,/opt/thpc,/storage2/fs1,/cm,/lib64/libmunge.so.2,/run/munge,/storage2/fs1,/scratch2/fs1,/storage1/fs1,/rdcw/fs1,/rdcw/fs2' #SBATCH --container-workdir='/storage1/fs1/gunnar/Active/tickets/itdev-37258/intel-oneapi/sbatch/c2-thpc' # load modules module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0 # export env vars export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so # C Tests echo "Running C test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c mpicc -o mpi_hello_world_c ./mpi_hello.c srun --mpi=pmi2 ./mpi_hello_world_c # Fortran Tests echo "Running Fortran test from: $(pwd)" wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f mpifc -o mpi_hello_world_f ./mpi_hello.f srun --mpi=pmi2 ./mpi_hello_world_f

  1. Execute the sbatch job using sbatch

    sbatch /path/to/sbatch_file
    [gunnar@c2-login-002 ~]$ sbatch ./intel-oneapi-helloworld.slurm Submitted batch job 149330

  2. Monitor job output in the .out file within the current working directory.

    tail -f <job_name>-<job_id>.out
    [gunnar@c2-login-002 ~]$ tail -f ./intel-oneapi-helloworld-149330.out 2025-09-05 13:02:57 (181 MB/s) - ‘mpi_hello.f’ saved [913/913] 2025-09-05 13:02:57 (193 MB/s) - ‘mpi_hello.f’ saved [913/913] Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4

  3. There will be a section for the C code and a section for the Fortran code

    [gunnar@c2-login-002 ~]$ grep -i -E 'running|hello' ./intel-oneapi-helloworld-149330.out Running C test from: /home/gunnar Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4 Running Fortran test from: /home/gunnar Hello from task 1 on c2-node-062 Hello from task 3 on c2-node-064 Hello from task 2 on c2-node-063 Hello from task 0 on c2-node-061 MASTER: Number of MPI tasks is: 4

Open OnDemand

Currently limited to only single-task single-node jobs. (--ntasks=1 --nodes=1)
Unable to launch multi-node jobs at this time.

  1. Start a C2-THPC job in OOD

All following commands are to be run INSIDE the interactive OOD job.

  1. Load the relevant modules

    module load ris module load intel-oneapi-compilers/2025.0.0 intel-oneapi-mpi/2021.14.0

  2. Set I_MPI_PMI_LIBRARY to a valid PMI library

    export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi2.so

Intel OneAPI does not have support for PMIX at this time.
PMI2 is preferred over PMI1 for better scaling performance.

  1. Download and compile sample code

    1. C

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.c && \ mpicc -o mpi_hello_world_c ./mpi_hello.c

    2. Fortran

      wget https://hpc-tutorials.llnl.gov/mpi/examples/mpi_hello.f && \ mpifc -o mpi_hello_world_f ./mpi_hello.f

  1. Execute sample hello_world binaries via mpirun within the Slurm job.

    1. C

      mpirun -np <num_cpus> ./mpi_hello_world_c
      [gunnar@c2-node-066 ~]$ mpirun -np 4 ./mpi_hello_world_c Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

    2. Fortran

      mpirun -np <num_cpus> ./mpi_hello_world_f
      [gunnar@c2-node-066 ~]$ mpirun -np 4 ./mpi_hello_world_f Hello from task 3 on c2-node-066! Hello from task 2 on c2-node-066! Hello from task 0 on c2-node-066! MASTER: Number of MPI tasks is: 4 Hello from task 1 on c2-node-066!

Due to OOD running all jobs in a single task (--ntasks=1) mpirun will attempt to always run with mpirun -np 1 unless otherwise set.

Versions

2025.0.0

Currently this is the only version of the Intel OneAPI base compilers suite installed into the x86_64 Spack environment directly supported by RIS AppEng.

Other packages, like intel-oneapi-mpi, show a considerably older release version as they are updated on a much slower cadence. This is to be expected.

More information around Intel OneAPI versions can be found on the Intel website.