IT recently discovered a problem with LSF jobs using Docker containers. We're about to push out a fix to this problem and you should know about some slight behavior changes.

ITDEV-5652 - Getting issue details... STATUS

What was broken?

We recently found that the Docker application for LSF doesn't correctly handle pipes. If you submitted a command like cat $FILE | cut -f 1, only the left side of the pipe (cat $FILE, in this example) would run in the container. Everything else would run outside of the container, directly on the LSF host itself! The resulting process tree would look something like this:

\_ /opt/lsf9/9.1/linux2.6-glibc2.3-x86_64/etc/res ...
    \_ /bin/sh /home/vagrant/.lsbatch/1510946154.1543
        \_ /opt/lsf9/9.1/linux2.6-glibc2.3-x86_64/bin/docker_run.py cat $FILE
            \_ docker run ... bunch of options here ... $IMAGE cat $FILE
        \_ cut -f 1

So cat $FILE would run in the container. The cut command would run outside the container and would just receive the output of the docker_run.py script, which manages pulling, running, and removing the Docker image. And if the right side of the pipe wasn't something simple like cut, it would fail outright because our Docker HPC blades don't have much software installed on them beyond Docker, LSF, and GPFS.

What's changing?

Basically we're just sticking quotes around the submitted command in an LSF configuration file and making sure we properly escape shell commands at a few other levels in LSF. But there will be some changes in behavior that you'll notice!

Once this change goes live, if your command relies on any shell features (pipes, redirects, operators like '||' or '&&') you'll need to do one of two things:

Stick your command in a small script and have your job execute that script
Wrap your command in /bin/bash -c

For example:

> bsub -q research-hpc -a 'docker(ubuntu)' /gscuser/bdericks/script.sh
> bsub -q research-hpc -a 'docker(ubuntu)' /bin/bash -c "cat $FILE | cut -f 1 > /gscuser/bdericks/output.txt"

If you don't do the above and you attempt to use shell features, they won't be recognized as such. For example, if you submitted a command like sleep 15 | sleep 30, you'll see a failure like the below. The '|' character is not recognized as a shell pipe and is passed as a regular argument to sleep, which it doesn't recognize.

sleep: invalid time interval '|'
sleep: invalid time interval 'sleep'
Try 'sleep --help' for more information.

Why the change?

It's important that the full submitted command be run inside the container. At best you get confusing behavior. At worst there's security implications because MGI IT has restricted direct access to our Docker HPC hosts and this erroneous behavior allows users to get around that.

We tried several solutions that would preserve the ability to directly submit commands using shell features, but they each fell short in one way or another:

If we add a /bin/bash -c to the front of every submitted command, that breaks any images that use an entrypoint.
It's possible to detect if an image has an entry point using docker inspect. That does make it possible to preserve the entrypoint and wrap the submitted command in /bin/bash -c, but...
We can't assume that the Docker image has /bin/bash. Many minimal containers may not.
/bin/sh is often a symlink. And it points to different shells across different OS. That seemed like a mess for us to manage.

Ultimately, the user submitting the job should know if their image has an entrypoint and what shells are available.

Questions and Problems

If you notice any problems with your LSF jobs after this change goes out, make an IT Support Desk issue at https://servicedesk.gsc.wustl.edu and we'll help you out as soon as we can!

Browser not supported