Docker Wrapper Environment Variables
Features of the docker
application (a.k.a. the docker wrapper) are accessed by setting environment variables before executing bsub
. All the examples below show how to set the values inline with submitting the job, and multiples of these settings are given by including all of them on the same command-line:
LSF_DOCKER_OPTTION_1=value LSF_DOCKER_OPTION_2="value with spaces" bsub ...
Or they can also be set within the shell by “exporting” them, essentially making them apply to every command that follows:
[user@compute1-client-1 ~]$ export LSF_DOCKER_OPTION_1=value [user@compute1-client-1 ~]$ export LSF_DOCKER_OPTION_2="value with spaces" [user@compute1-client-1 ~]$ bsub ...
LSF_DOCKER_ADD_HOST
Adds one more more host -> IPaddress mappings to the docker container. The value of LSF_DOCKER_ADD_HOST
is a space-separated list of mappings in the same format as the --add-host
argument of docker run
, ie. hostname:IPaddress
LSF_DOCKER_ADD_HOST="host_a:192.168.1.2 host_b:172.16.10.2" bsub ...
LSF_DOCKER_ADD_HOST_FILE
Specify one more more space-separated files containing host -> IPaddress mappings. These files should have one mapping per line in the same format used by LSF_DOCKER_ADD_HOST:
host_a:192.168.1.2 host_b:172.16.10.2
These file names are then included in the LSF_DOCKER_ADD_HOST_FILE
variable:
LSF_DOCKER_ADD_HOST_FILE="file_a file_b" bsub ...
LSF_DOCKER_CGROUP
Set the value for the --cgroup-parent
argument to docker run
. There is no mechanism for creating custom cgroups at this time, so this should not be used unless specifically directed to by RIS.
LSF_DOCKER_ENTRYPOINT
Overrides the container’s default ENTRYPOINT
, and becomes the --entrypoint
argument to docker run
LSF_DOCKER_ENTRYPOINT=/bin/bash bsub ...
LSF_DOCKER_ENV_FILE
Used to set envrionment variables within the running container. Expects a space-separated list of file names containing environment variables. The files should have one variable per line, and be formatted as variable=value
:
SOME_VARIABLE=value ANOTHER_VARIABLE=value with spaces
To make use of these files, set the LSF_DOCKER_ENV_FILE
and submit the job:
LSF_DOCKER_ENV_FILE="env_file1 env_file2" bsub -q general -a 'docker(...)' ...
The file names are then used as --env-file
arguments to docker run
.
LSF_DOCKER_IPC
Control the --ipc
argument to docker run
. This will become useful when building MPI applications.
export LSF_DOCKER_IPC=host bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
LSF_DOCKER_NETWORK
Make use of docker’s --network
argument. Most notablly for --network=host
. See the Docker networking tutorial.
export LSF_DOCKER_NETWORK=host bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
LSF_DOCKER_PORTS
Specify network ports to expose from your docker container. Note that these must come from a range of “approved ports” (8000-8999) and should be “reserved” by the job scheduler. Here we map port 80 inside the container to port 8001 on the execution node, and we “reserve” port 8001 on that execution node through the scheduler.
export LSF_DOCKER_PORTS='8001:80' bsub -q general -R 'select[port8001=1]' -a 'docker(httpd)' /usr/local/bin/httpd-foreground
The LSF_DOCKER_PORTS
variable works with most expressions accepted by the -p
option of docker run
:
-
8001
: Expose a single port -
8001:80
: Expose port 8001 and forward it to port 80 within the container -
8001-8010
: Expose a range of ports -
8001-8010:2001-2010
: Expose a range of forwarded ports -
8001/tcp
: Expose only TCP port 8001 -
8001-8010/udp
: Expose a range of UDP ports
Additionally, more than one port may be exposed by separating each request with a space. For example:
LSF_DOCKER_PORTS='8000 8500 8600-8700`
See also Exposing Ports from Within Containers.
Note: LSF_DOCKER_PORTS is affected by the LSF_DOCKER_NETWORK type, and does not work with “host” type networking.
LSF_DOCKER_PRESERVE_ENVIRONMENT
This is a boolean value instructing the job launcher to either preserve your shell environment or not. Valid values are the strings true
and false
. The default value is true
.
When false, the container will have a minimal environment with only a few variables, including HOSTNAME
set to the exec host the job runs on, and those variables defined in the container’s Dockerfile with ENV
.
When true, the container will inherit most environment variables set when the job is submitted, except for a few:
-
HOSTNAME - becomes the exec host’s hostname
-
LSB_INTERACTIVE - removed within an interactive job so jobs started within that job are not interactive by default
-
values containing newlines - these variables are removed due to a bug in docker’s handling of environment variables
export LSF_DOCKER_PRESERVE_ENVIRONMENT=false bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
LSF_DOCKER_RUN_LOGLEVEL
Set the log level for the wrapper script:
export LSF_DOCKER_RUN_LOGLEVEL=DEBUG bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
Valid values are, in descending order of verbosity: CRITICAL, ERROR, WARNING, INFO, DEBUG
LSF_DOCKER_SHM_SIZE
Control the --shm-size
argument to docker run
. This will become useful when building MPI applications.
export LSF_DOCKER_SHM_SIZE=4g bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
LSF_DOCKER_USE_LUCID_AUTH
Getting authentication through libnss-sss working for containers based on Ubuntu Lucid Lynx (10.04) requires extra files and devices from the host system to be mounted in the container. This option should not be used unless you are having trouble getting authentication working on a container based on Ubuntu Lucid.
LSF_DOCKER_USE_LUCID_AUTH=y bsub ...
LSF_DOCKER_WORKDIR
The default initial working directory within the container is the same as the working directory when the job was submitted. This can be overridden with LSF_DOCKER_WORKDIR
and becomes the -w
option to docker run
, and must be a directory accessible within the container.
LSF_DOCKER_WORKDIR=/path/within/container bsub ...
LSF_DOCKER_VOLUMES
This is a space separated list of filesystem locations to pass into your docker container by means of the --mount
flag. The format is src:dst
, where src
means the filesystem location outside the container should be mapped to dst
inside the container. (See also --mount
at Docker Volumes.)
export LSF_DOCKER_VOLUMES="/storage1/fs1/${STORAGE_ALLOCATION}:/storage1/fs1/${STORAGE_ALLOCATION} /scratch1/fs1/${COMPUTE_ALLOCATION}:/scratch1/fs1/${COMPUTE_ALLOCATION}" bsub -Is -G ${group_name} -q general-interactive -a 'docker(ubuntu)' /bin/bash
Elements in LSF_DOCKER_VOLUMES must be directories. Docker would allow you to pass in files, such that the following might be expected to work:
LSF_DOCKER_VOLUMES=”$HOME/etc/myfile:/etc/somefile”
But the RIS environment explicitly prevents this due to a security vulnerability.
A volume can be mounted read-only by appending :ro
to any of the mounts, for example: src:dst:ro
.
LSF_DOCKER_RUN_RAW_ARGS
This variable tells bsub to use the command or entrypoint within the Docker image instead of requesting the command or entrypoint from the user as is the regular interaction without this variable.
This means that Docker images like the basic hello-world Docker image can be run on the Compute Platform.
The command also makes use of LSB_DOCKER_PLACE_HOLDER
which is, as the name suggests, just a placeholder as bsub still needs an input to run. This is NOT passed as a command to the Docker container.
Example Command:
LSF_DOCKER_RUN_RAW_ARGS=1 bsub -Is -G ${group_name} -q general-interactive -a 'docker(hello-world)' LSB_DOCKER_PLACE_HOLDER
Output:
Job <897631> is submitted to queue <qa-interactive>. <<Waiting for dispatch ...>> <<Starting on compute1-exec-83.ris.wustl.edu>> Using default tag: latest latest: Pulling from library/hello-world Digest: sha256:ffb13da98453e0f04d33a6eee5bb8e46ee50d08ebe17735fc0779d0349e889e9 Status: Image is up to date for hello-world:latest docker.io/library/hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/