/
C2 Spack v1.1+

You can use our service desk portal for getting RIS support. RIS also offers 15 min. virtual office hours session Mon-Thru..

C2 Spack v1.1+

This page exists to detail how a user would be able to use the RIS provided Spack installation under /opt/thpc to build their own custom software. The software can be stored in a user’s home2, a shared storage2, or a shared scratch2 location.

Initial Setup

Load modules

  1. Connect to a RIS C2 Login node

    ssh c2-login-001.ris.wustl.edu

  2. Load the RIS domain module followed by the Spack module

    module load ris module load spack/v1.1

Configuring Spack

Spack can be configured by end-users beyond what RIS has provided by default. This can be done by layering or entirely overwriting RIS configurations by writing out specific yaml files placed in $HOME/.spack/

For example to override modules configuration one would create a new modules.yaml file at $HOME/.spack/modules.yaml into which they can write their custom configuration(s).

To see the current full working configuration of Spack, including how any custom configuration has been layered, run the following:

spack config blame

The configuration output can be restricted to specific sections for modules or packages like so:

spack config blame modules # or spack config blame packages

See all config sections available to print like so:

spack config blame --help

For more information on Spack configuration scopes and how to use them please see the official documentation: https://spack.readthedocs.io/en/latest/configuration.html

 

Installing Software

Home Directory

home2 is limited in size to around 50GiB per user.

Without any additional configuration Spack will default to installing software and modules to a directory in a user’s home directory. This is the intended workflow for small incidental tools that do not necessitate the creation of an entire Spack Environment for reproduction and/or sharing purposes. Applications such as: tree, vim, gcc, miniconda, etc

  1. Launch an interactive slurm job.

    srun -A compute2-account -c 8 -p general-interactive --pty /bin/bash

  2. Install a software of choice. tree shown below

    spack install tree

  3. Add the modules to Lmod. Currently only lua based modules are built by default.

    # lua modules module use $HOME/.spack/modules/lmod/Core

  4. Confirm that you now see your modules displayed with module avail. module avail tree used below. All dependencies hidden unless using module --show_hidden avail

    image-20260213-185016.png

Shared Location - Informal

If intending to produce a set of packages via Spack that are to be shared and/or further modified by other members of a lab or some working group it is advised to utilize Spack Environments.

Such an environment can contain a grouping of packages distinct from those installed to a user’s home directory. With different, or even conflicting, requirements. Generally these environments are installed to RIS Storage or Scratch clusters.

storage2 or storage3 are recommended to hold user Spack environments due to storage retention policy

scratch2 can be used for increased performance but this requires routine rebuilds of software and/or modules due to the default retention policies on that location

storage1 can be used but is not recommended by RIS at this time.

  1. Launch an interactive slurm job.

    srun -A compute2-account -c 8 -p general-interactive --pty /bin/bash

  2. Create a new directory on storage2 or storage3 to act as the root of the new Spack environment. This location should be owned and administered by you or your PI.

    mkdir -p /storageN/fs1/allocation_name/Active/spack/myenv1

  3. In the newly created directory for the Spack environment create the following child directories as well: environment,software,modules,license,views

    mkdir -p /storageN/fs1/allocation_name/Active/spack/myenv1/{environment,software,modules,license,views}

  4. In the newly created environment directory now create the spack.yaml file for the environment.

    vi /storageN/fs1/allocation_name/Active/spack/myenv1/environment/spack.yaml
    spack: specs: # List of all Spack software(s) and tweaks to be installed - gcc@15+binutils # latest GCC-15.x with GNU Assembler - tree # latest satisfiable version of tree - vim # latest satisfiable version of vim view: false config: install_tree: root: /storageN/fs1/allocation_name/Active/spack/myenv1/software/spack license_dir: /storageN/fs1/allocation_name/Active/spack/myenv1/licenses modules: default: roots: lmod: /storageN/fs1/allocation_name/Active/spack/myenv1/modules/spack lmod: # optionally add additional custom non-RIS compilers # RIS default is 'gcc' and will be appended core_compilers: []

Update the spack.yaml file to build your environment as needed:

  1. spack.specs: update with a list of all the packages you wish to install

  2. spack.config.install_tree.root: update with the path to the software/spack directory created for your environment in Step3

  3. spack.config.license_dir: update with the path to the licenses directory created for your environment in Step3

  4. spack.modules.default.roots.lmod:update with the path to the modules/spack directory created for your environment in Step3

  5. Additional tweaks and changes not required but can be made as user needs demand

  1. Add the new environment to your list of “tracked” Spack environments. Be sure to provide a --name <name> value otherwise the directory above spack.yaml will set the name.

    spack env track --name myenv1 /storageN/fs1/allocation_name/Active/spack/myenv1/environment

  2. Install the newly configured Spack environment

    1. Concretize the changes to spack.yaml to build a spack.lock file in the same directory

      spack --env myenv1 concretize --fresh --force

    2. Install the software based on the generated spack.lock file

      spack --env myenv1 install

    3. Regenerate any modules if any currently exist as Spack will NOT overwrite them by default.

      spack --env myenv1 module lmod refresh --delete-tree --yes-to-all

  3. Add the newly built software to your Lmod modulepath and confirm it shows up now

    module use /storageN/fs1/allocation_name/Active/spack/myenv1/modules/spack/Core

    1. If using a compiler other than that supplied by RIS you may see the Core directory match the name and version of the compiler used to build the software. For example gcc-15.1.0. To fix this you must add the compiler to the list of core_compilers in your environment config.

 

Domain Space - Formal

Formalized shared “domain spaces” can be setup for PIs, labs, or other working groups upon request. The purpose of which is to give groups a means to formally define and share software amongst themselves or other groups. These domain spaces are currently held under /opt/thpc and would include all modules, softwares, license files, and Spack configurations required to operate and/or reproduce the software contained within.

Once a domain is created PIs are encouraged to create as many Spack environments as needed following the pattern below.

/opt/thpc/linux/x86_64 ├── ${DOMAIN_NAME} │ └── ${ENV_NAME} │ └── {environment,license,modules,software,views} └── artsci # Domain name └── x86_64 # Spack environment name ├── environment # contains `spack.yaml` and `spack.lock` ├── license # contains licenses in nested app dirs ├── modules │ ├── spack # contains all modules produced by Spack │ └── other # contains any modules NOT from Spack ├── software │ ├── spack # Spack software install_root │ └── other # software NOT from Spack but considered within this env └── views # holds Spack views if enabled

To request a new domain be constructed or to update domain contact information please reach out to RIS Support.

 

Setup

Once a domain is created to hold your Spack environments we need a directory for each respective environment.

The following examples will assume that the domain is artsci, already exists, and that I am making an environment named myenv2. Please update the commands with your own relative paths as needed

  1. Create the following locations for your domain specific Spack environment if they do not already exist.

    1. config:environments_root

      mkdir -p /opt/thpc/linux/x86_64/artsci/myenv2/environment

    2. config:license_dir

      mkdir -p /opt/thpc/linux/x86_64/artsci/myenv2/license

    3. config:install_tree:root

      mkdir -p /opt/thpc/linux/x86_64/artsci/myenv2/software/spack

    4. modules:default:roots:lmod

      mkdir -p /opt/thpc/linux/x86_64/artsci/myenv2/modules/spack

  2. Create a new spack.yaml file in the newly created environment directory. If you are updating an existing environment, you should modify the existing spack.yaml.

    vi /opt/thpc/linux/x86_64/artsci/myenv2/environment/spack.yaml
    spack: specs: # List of all Spack software(s) to be installed - gcc@15+binutils # latest GCC-15.x with GNU Assembler variant enabled - tree # latest satisfiable version of tree - vim # latest satisfiable version of vim view: false # disabled view generation by default for faster installation config: install_tree: root: /opt/thpc/linux/x86_64/artsci/myenv2/software license_dir: /opt/thpc/linux/x86_64/artsci/myenv2/licenses modules: default: roots: lmod: /opt/thpc/linux/x86_64/artsci/myenv2/modules/spack lmod: # optionally add additional custom non-RIS compilers # RIS default is 'gcc-13.4.0' and will be appended as 'gcc@=13.4.0' core_compilers: []

Update the spack.yaml file to build your environment as needed:

  1. spack.specs: update with a list of all the packages you wish to install

  2. spack.config.install_tree.root: update with the path to the software/spack directory created for your environment in Step3

  3. spack.config.license_dir: update with the path to the licenses directory created for your environment in Step3

  4. spack.modules.default.roots.lmod:update with the path to the modules/spack directory created for your environment in Step3

  5. Additional tweaks and changes not required but can be made as user needs demand

  1. To see the environment when you run spack env list you must add it to the tracked environments. Do not activate the environment after tracking it.

    # environment name defaults to dir name unless `--name <NAME>` provided spack env track --name myenv2 /opt/thpc/linux/x86_64/artsci/myenv2/environment

 

Interactive

  1. Launch an interactive slurm job.

    srun -A compute2-account -c 8 -p general-interactive --pty /bin/bash

  2. Load the ris domain and spack modules

    module load ris module load spack/v1.1

  3. Install the environment

    1. Concretize the environment build a spack.lock based on any changes to the spack.yaml

      spack --env myenv2 concretize --fresh --force

    2. Install the environment as it has been concretized in spack.lock

      spack --env myenv2 install

    3. Rebuild any modules that already existed before reinstall.

      spack --env myenv2 module lmod refresh --delete-tree --yes-to-all

Non-Interactive

  1. Create sbatch file to run installation across nodes named batch-install.sh

    #!/bin/bash #SBATCH -A compute2-account #SBATCH --partition=general-cpu #SBATCH --nodes=8 #SBATCH --cpus-per-task=8 #SBATCH --time=12:00:00 module purge module load ris module load slurm/23.02.5 spack/v1.1 spack --env myenv2 concretize --fresh --force srun spack --env myenv2 install spack --env myenv2 module lmod refresh --delete-tree --yes-to-all

  2. Execute the sbatch file

    sbatch ./batch-install.sh

Repos

Spack has the concept of both local and remote repositories. In versions prior to 1.0 local was the only properly supported option. Following the release of Spack 1.0 the default is to use a Git based repository.

This has allowed for the decoupling of the default builtin Spack repository from the system installation of Spack. Such that each user of Spack now holds a local copy of the builtin package git repository in their home directory.

Meaning end-users can now easily directly edit their own local copy of a Spack package’s python install files with spack edit builtin.<package_name>

 

Adding a Repo

Adding a “Spack Repo” in this way will see all changes written to the “User Scoped Config” at $HOME/.spack/repos.yaml

For more information please see the Spack Documentation here: https://spack.readthedocs.io/en/v1.1.1/repositories.html#git-based-repositories

Builtin Repo

This “Spack Repo” is already be added by default when using the RIS Spack module.

  1. Add the repository’s main branch via GitHub url

    spack repo add --name myrepo https://github.com/spack/spack-packages.git

  2. Confirm repo now shows up

    spack repo list

  3. Update local cache as needed

    spack repo update

RIS Remote Repo

The washu_it_ris “Spack monorepo” used in this example is already included by default when using the RIS Spack module.

  1. Add the repository

    spack repo add --name compute2-spack-packages https://github.com/washu-it-ris/compute2-spack-packages.git
    [gunnar@c2-login-001 ~]# spack repo add --name compute2-spack-packages https://github.com/washu-it-ris/compute2-spack-packages.git remote: Enumerating objects: 21, done. remote: Counting objects: 100% (21/21), done. remote: Compressing objects: 100% (17/17), done. remote: Total 21 (delta 1), reused 14 (delta 0), pack-reused 0 (from 0) ==> Added repo to config with name 'compute2-spack-packages'.

  2. Confirm repos now show up

    spack repo list
    [gunnar@c2-login-001 ~]# spack repo list [+] washu_it_ris.ris v2.0 /root/.spack/package_repos/aozlvq7/repos/spack_repo/washu_it_ris/ris [+] washu_it_ris.artsci v2.0 /root/.spack/package_repos/aozlvq7/repos/spack_repo/washu_it_ris/artsci [+] builtin v2.2 /root/.spack/package_repos/fncqgg4/repos/spack_repo/builtin

Update a Repo

Sync Remote Changes

  1. Update local per-user repo cache

    spack repo update

Override default Branch

We will be using the builtin repository for this example. This allows a user to switch to newer or older instances of the default repo than is offered by RIS.

The same steps can be applied to any other repositories as needed.

  1. Identify the current repository that you wish to redirect to another Git branch

    spack repo list
    [gunnar@c2-login-001 ~]$ spack repo list [+] washu_it_ris.ris v2.0 /home/gunnar/.spack/package_repos/aozlvq7/repos/spack_repo/washu_it_ris/ris [+] washu_it_ris.artsci v2.0 /home/gunnar/.spack/package_repos/aozlvq7/repos/spack_repo/washu_it_ris/artsci [+] builtin v2.2 /home/gunnar/.spack/package_repos/fncqgg4/repos/spack_repo/builtin

  2. Examine the location and state of the configuration for the repository you wish to change the branch of

    spack config blame repos
    [gunnar@c2-login-001 ~]$ spack config blame repos --- repos: /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:3 compute2-spack-packages: /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:4 git: https://github.com/washu-it-ris/compute2-spack-packages.git /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:5 branch: main /opt/thpc/linux/software/spack/v1.1/etc/spack/defaults/base/repos.yaml:14 builtin: /opt/thpc/linux/software/spack/v1.1/etc/spack/defaults/base/repos.yaml:15 git: https://github.com/spack/spack-packages.git /opt/thpc/linux/software/spack/v1.1/etc/spack/defaults/base/repos.yaml:16 branch: releases/v2025.11

  3. Overwrite the configuration to use the newer v2026.03.0 branch from the upstream Spack project by setting new values in $HOME/.spack/repos.yaml

    vi ~/.spack/repos.yaml
    # ~/.spack/repos.yaml repos: builtin: branch: v2026.03.0

  4. If successful you should now see the branch value changed for the builtin repo

    spack config blame repos
    [gunnar@c2-login-001 ~]$ spack config blame repos --- repos: /home/gunnar/.spack/repos.yaml:2 builtin: /home/gunnar/.spack/repos.yaml:3 branch: v2026.03.0 /opt/thpc/linux/software/spack/v1.1/etc/spack/defaults/base/repos.yaml:15 git: https://github.com/spack/spack-packages.git /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:3 compute2-spack-packages: /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:4 git: https://github.com/washu-it-ris/compute2-spack-packages.git /opt/thpc/linux/software/spack/v1.1/etc/spack/site/repos.yaml:5 branch: main

  5. You can now run the spack repo update command to sync up your local copy

    spack repo update
    [gunnar@c2-login-001 ~]$ spack repo update remote: Enumerating objects: 504631, done. remote: Counting objects: 100% (504631/504631), done. remote: Compressing objects: 100% (149596/149596), done. remote: Total 495928 (delta 212635), reused 494193 (delta 211290), pack-reused 0 (from 0) ==> builtin: Updated sucessfully. ==> compute2-spack-packages: Already up to date.

Edit a Package

  1. Set the editor that you wish to use if you haven’t already

    export EDITOR='vim'

  2. Edit the package with the syntax <repo_name>.<package_name>

    1. matlab within the builtin repo

      spack edit builtin.matlab

    2. matlab within the ris repo

      spack edit washu_it_ris.ris.matlab

    3. matlab within the default repo (first repo shown in spack repo list)

      spack edit matlab