Apptainer Example: svFSIplus

svFSIplus is a C++ rework of svFSI. svFSI is a multi-physics finite element solver designed for computational modeling of the cardiovascular system.

As with many software projects for research, svFSI is a moving target for system administrators - there is a base tension between the ability to provide stable environments against the needs of cutting-edge software. Containers help bridge that gap - allowing users to run software with particular requirements on almost any system.

The McKelvey Engineering cluster supports Singularity/Apptainer, and below is how a svFSIplus container was built for that cluster.

One of the easiest ways to find out how to build a particular piece of software in the current year is by finding if the project publishes a Dockerfile for creating Docker containers. Dockerfiles are convertible to Apptainer recipes, but instead, here, we go through the steps manually to produce our file. If we needed to rebuild this software or container often, then converting the Dockerfile would be warranted.

svFSIplus’s Dockerfile is found here:

https://github.com/SimVascular/svFSIplus/blob/main/Docker/ubuntu22/dockerfile#L4

This one is broken up into sections by each prerequisite in order of need. For this document, we’re going to gloss over much of the Dockerfile structure, especially the parts we don’t need for this particular exercise, and highlight the important parts by line number as shown on the Github page.

This document was written with the Dockerfile as of 7/26/24 in that repository.

Line 5: FROM ubuntu:22.04 AS buildcmake

This tells us our base image. On build.engr.wustl.edu, we’ll create an Apptainer sandbox with that same base:

sudo apptainer build --sandbox ubuntu22-svfsiplus docker://ubuntu:22.04

Once that processes, we enter the sandbox in writable mode:

sudo apptainer shell --writable ubuntu22-svfsiplus

Now we’re ready to start prepping the base of the container. First, we look through the Dockerfile for (since this is Ubuntu) lines about “apt-get update” and “apt-get install”. We find those on Line 17. and it so happens they are the same for each software section. We’ll go ahead and run those in the container:

Before we continue, we need to create a build area. We’re going to do that for this exercise in the container itself. That is where we will download and compile the software. When we’re done, we’ll move it out of the container tree to save space.

cmake

We find the instructions for cmake starting on Line 29. There is a ${CMAKE_VERSION} variable there, which will happen with many of these pre-reqs. We can find that defined on Line 11. We’ll put that together and download and unpack the software:

To build cmake:

openmpi

Line 73

VTK

Line 119

Boost

Line 190

Lapack

Line 233

HDF5

Line 330

Hypre

Line 387

Trilinos

Line 436

PETsc

Line 522

svFSIplus

This is not part of the Dockerfile - they’ve constructed that as a base image ready to compile svFSIplus. We’re going to complete that here. The instructions for svFSIplus are here:
https://simvascular.github.io/svFSIplus/index.html

Testing

Before we exit the container, we can test. The full documentation shows how to run the full suite of tests, which we’d do if we were running the container properly, with some modification. We’ll pick a small one here to do manually to prove we compiled things OK.

Cleanup and Packaging

Before we go, we should clean up the APT cache and database, where packages we installed on the first step were downloaded:

We can then exit the container, by typing “exit”. That should land you back at the same terminal but outside the container. We left a large compilation tree inside the container under /usr/local/sv. We can keep it, but remove it from the container, or delete it. This example will move it for now, in case it needs to be referenced later.

We then want to compile up the container into a SIF file.

apptainer build svFSIplus.sif ubuntu-svfsiplus

That leaves us with the svFSIplus.sif file, which we can copy someplace appropriate for our lab or ourselves. This one is only 773MB, so it could reside in our home directory (15GB quota) without taking up too much space.

To use the container, we’d simply call the binary we left in there, referencing the container wherever it resides:

apptainer run /project/engineering/svfsiplus/svfsiplus.sif svFSIplus svFSI.xml

Or, since svFSIplus uses OpenMPI for multicore:

apptainer run /project/engineering/svfsiplus/svfsiplus.sif mpirun -n 32 svFSIplus svFSI.xml

That same executable line can be used in job script submissions in place of however you’d usually run your executable. That holds true for compiled binaries, or containerized Python environments, or most everything else.

Sample Job File

The above file runs a job that:

• Puts output in a file name svfsiplus.123456, where 123456 is the job number assigned (-0)

• Sends an email when the job is done (-N)

• Names the job svFSIplusJob in “bjobs” output (-J)

• Selects a node with 128GB of available RAM, avoiding hosts with GPUs and keeping all the slots on a single host (-R)

• Selects a node on the cpu-compute queue (-q)

• Selects a node with 32 free cores (-n)

It uses the environment variable $LSB_DJOB_NUMPROC so that you only have to change the number of cpu slots requested in one place, at the top of the file in the #BSUB -n line.