How Docker Relates to the HPC Environment

Why Docker?

In the past, HPC environments were built upon “static operating system images”, which is to say that each execution node of a cluster had the same Operating System Image, with a set of applications that were curated and installed by the managing IT team. Each application was therefore exposed to every application’s “dependency tree”. For example, if Application A required Library Z version 1, but Application B required Library Z version 2, the applications conflicted with each other. Various methods were devised over the years to try to isolate application environments from each other. The development of “modules” that would use environment variables and shared filesystems was an attempt to solve this problem. In the end, the modules and the environments needed to be built by the cluster managers. End users had limited ability, if any at all, to deploy the software they wanted “on the fly”.

...

Docker allows users to build their own software environments independently of anyone else. The cluster management team no longer needs to be the gatekeeper controlling the available software.

What is Docker?

As described in What is a Container?,

...

A container image is built and pushed to a container registry for later use. In the RIS Compute Service, a user submits a job that is managed by the IBM Spectrum LSF job scheduler that, when executed, pulls the docker container image to an execution node where it is executed on behalf of the user.

The Relationship Between Users, LSF, and Docker

• The following diagram represents relationships between the user and portions of the cluster:

...

• The important thing to know here is that the user is using Docker, but not directly. There exists both a “job scheduler” and a “wrapper script” between the user and the docker run command.

• This is important to know because many of the features RIS offers users are mitigated by this wrapper script, and some are disallowed, usually for security reasons.

• Most features of the docker wrapper are exposed through environment variables. See Docker Wrapper Environment Variables for the list of them and their use.

Public vs Private Registries

Sometimes a user does not want, or is forbidden, to put code into a public registry. RIS suggests use of Docker Hub’s private registries as WashU does not currently have a private container registry. See Build Images Using Compute for how to log into and build using private registries on compute.

Locate Preexisting Docker Images

There are many sources of preexisting Docker images, primarily within Docker Hub’s public registry. Before building an image from scratch it is recommended you search for an image to meet your requirements.

Build Images Using Compute

If you are unable to locate an image on a public registry that meets all of your needs, it will need to either be built from scratch or by building upon an existing container and pushing for future use.

RIS recommends building Docker images on a workstation or other computer you have local access to as it makes debugging the build process easier. However, some build processes may require more resources than you have available locally. For these situations, the compute cluster can be used.

The docker_build LSF application accepts all the same arguments as the command-line docker build, except for Windows-specific options such as --security-opt. It works with build contexts that are subdirectories or URLs that are git repos or tarballs.

1. Log Into Public or Private Registry

The docker_build application uses various environment variables to interact with public and private repositories. Registry credentials are stored in the file $HOME/.docker/config.json. Rather than passing the password in as an environment variable, it’s a better practice to first log into the registry interactively with LSB_DOCKER_LOGIN_ONLY enabled. The credentials will automatically be populated into the config file and used when pushing to that registry.

If the docker login process would ask for a password, be sure to run the login build with bsub’s interactive flag, for example:

• Docker Hub:

  Code Block
  > LSB_DOCKER_LOGIN_ONLY=1 \ bsub -G ${group_name} -q general-interactive -Is -a 'docker_build' -- .

• Other repository:

  Code Block
  > LSB_DOCKER_LOGIN_ONLY=1 \ LSB_DOCKER_LOGIN_SERVER=repo.example.com \ bsub -G ${group_name} -q general-interactive -Is -a 'docker_build' -- .

2. Build and Push Image

Once logged in, repositories accessible with those credentials, both public and private, will be accessible.

The docker_build LSF application accepts all the same arguments as the command-line docker build, except for Windows-specific options such as --security-opt. It works with build contexts that are subdirectories or URLs that are git repos or tarballs.

For example, this command will submit a job to build a container based on files in the ‘my_container’ subdirectory, tag it with 1.1.4 and 1.1. All the tags that appear, both as a “docker_build” argument and “–tag” option, must also be valid to push to with “docker push”, and all tags are pushed after the build completes.

Code Block
> bsub -G ${group_name} -q general-interactive -Is -a 'docker_build(repo.example.com/repo_username/example_container_name:1.1.4)' -- --tag repo.example.com/repo_username/example_container_name:1.1 example_container_build_directory

Take note of the -- that separates the arguments to bsub from the arguments to “docker build”. Without it, bsub will try to interpret all the arguments for itself and generate an error. Also, the “build context” directory path must be the final argument. Other arguments recognized by “docker build” must appear before the build context argument.

This command will submit a job where the Dockerfile is in a non-standard place and is not named Dockerfile

Code Block
> bsub -G ${group_name} -q general-interactive -Is -a 'docker_build(repo.example.com/repo_username/example_container_name:latest)' -- -f /path/to/the/NonStandardDockerfile example_container_build_directory

Docker Images Must Include /bin/sh

By default, the IBM Spectrum LSF software job scheduler requires that the Docker containers launched have a /bin/sh present. Users may observe that Docker uses hello-world as an example in documentation. This container is an example of one that does not include a /bin/sh. Launching a container that does not supply /bin/sh results in a “no such file or directory” error:

...

This can be circumvented by the use of an LSF variable so that Docker image that do not have supply /bin/sh can be run on the Compute Platform. Please see our LSF env variable documentation for more information.

The Job Execution Wrapper Script

The wrapper process that builds the docker run command does several things in order to construct a “safe” docker run command. The following is an example of job submission in order to demonstrate what the wrapper script is doing:

...

/usr/bin/docker                                            # The docker executable
--cidfile /tmp/lsf.compute1-lsf.job.23870.1580953423       # Write the container ID to this file
-u 1416339:1000070                                         # Run as this UID and GID
-v /opt/ibm/lsfsuite/lsf:/opt/ibm/lsfsuite/lsf             # Mount the job scheduler filesystem
-v /home/joe.user:/home/joe.user                           # Mount the current working directory
-w /home/joe.user/                                         # Set the "working directory"
-v /home/joe.user/.lsbatch:/home/joe.user/.lsbatch         # Mount the location of the job scripts
-v /tmp/23870.tmpdir:/tmp                                  # Mount LSF job's tempdir as /tmp
--rm                                                       # Remove the container when finished
-h compute1-exec-203.ris.wustl.edu                         # Set the container host name
--name=lsf_23870_337938                                    # Set the container name
--mount type=bind,source=/tmp/tmpysrUPG,target=/etc/passwd # Mount minimal /etc/passd
--mount type=bind,source=/tmp/tmpAV6MiF,target=/etc/group  # Mount minimal /etc/group
--group-add 1000070                                        # Pass in groups from AD
--group-add 1183521
--group-add 1262589
--mount type=bind,source=/var/lib/sss/pipes,target=/var/lib/sss/pipes  # Pass in sssd for IDM
--mount type=bind,source=/etc/nsswitch.conf,target=/etc/nsswitch.conf  # Pass in nsswitch for IDM
--mount type=bind,source=/dev/log,target=/dev/log                      # Pass in /dev/log for logging
--mount type=bind,source=/tmp/23870.tmpdir,target=/tmp/23870.tmpdir    # Pass in LSF job tmpdir
--security-opt label=user:user_u                           # Set SELinux contexts
--security-opt label=role:user_r
--security-opt label=type:user_t
--security-opt label=level:s0
--gpus all                                                 # Pass in GPU devices
--sig-proxy=false                                          # Do not proxy signals
-i                                                         # Be interactive
-t                                                         # Use a tty
-e LSF_NIOS_JOBSTATUS_INTERVAL=1                           # Set the LSF_NIOS_JOBSTATUS_INTERVAL
--cpu-shares 2                                             # Set default cpu shares
-m 4096m                                                   # Set default memory reservation
--memory-swap -1                                           # Disallow swapping to disk
--cap-drop=all                                             # Set allowed OS Capabilities
--cap-add=chown
--cap-add=kill
--cap-add=setpcap
--cap-add=IPC_LOCK
--cap-add=CAP_SYS_NICE
--security-opt=no-new-privileges
--device=/dev/infiniband/issm0                             # Add Infiniband devices
--device=/dev/infiniband/rdma_cm
--device=/dev/infiniband/ucm0
--device=/dev/infiniband/umad0
--device=/dev/infiniband/uverbs0
--ulimit memlock=-1                                        # Disable memory paging
--env-file=/tmp/lsf.compute1-lsf.job.23870.1580953423.env  # Set environment file
alpine                                                     # Use this container image
/home/joe.user/.lsbatch/1580953421.23870                   # Run this shell script via /bin/sh

What does all this mean?

Important take aways from the above exploration include:

...