Parabricks Quickstart
Image Details
-
Docker image hosted at nvcr.io/nvidia/clara/clara-parabricks:4.0.0-1
-
Official documentation for Parabricks version 4.0.0.
Getting Started
-
Connect to compute client.
ssh wustlkey@compute1-client-1.ris.wustl.edu
-
Prepare the computing environment before submitting a job.
# Use scratch file system for temp space export SCRATCH1=/scratch1/fs1/${COMPUTE_ALLOCATION} # Use Active storage for input and output data export STORAGE1=/storage1/fs1/${STORAGE_ALLOCATION}/Active # Use host level communications for the GPUs export LSF_DOCKER_NETWORK=host # Use debug flag when trying to figure out why your job failed to launch on the cluster #export LSF_DOCKER_RUN_LOGLEVEL=DEBUG # Use entry point since the parabricks container has other entrypoints but our cluster, by default, requires /bin/sh export LSF_DOCKER_ENTRYPOINT=/bin/sh # Create tmp dir export TMP_DIR=${SCRATCH1}"/parabricks-tmp" [ ! -d $TMP_DIR ] && mkdir $TMP_DIR
-
Submit job. Basic commands for use:
bsub -n 16 -M 64GB -R 'gpuhost rusage[mem=64GB] span[hosts=1]' -q general -gpu "num=1:j_exclusive=yes" -a 'docker(nvcr.io/nvidia/clara/clara-parabricks:4.0.0-1)' pbrun command options
Known Issues
-
- Parabricks relies on available GPU(s) noted with
NVIDIA_VISIBLE_DEVICES
which defaults to āallā regardless the quantity and -
device number of GPU(s) reserved at runtime. As such, there is a possibility the software will attempt to run on GPU(s) the job does not have access to. At this time it is advised to prepend
pbrun
with the following.for VAR in $(printenv | grep CUDA_VISIBLE_DEVICES); do export ${VAR/CUDA/NVIDIA}; done
- Parabricks relies on available GPU(s) noted with
Additional Information
-
- Cores (
-n
) and memory (-M
andmem
) may need to be adjusted depending on the data set used. -
-
1 GPU server should have 64GB CPU RAM, at least 16 CPU threads.
-
2 GPU server should have 100GB CPU RAM, at least 24 CPU threads.
-
4 GPU server should have 196GB CPU RAM, at least 32 CPU threads.
-
- Cores (
-
It is suggested to keep the GPUs at 4 and RAM at 196GB unless your data set is smaller than the 5GB test data set.
-
There is diminishing returns on using more GPUs on small data sets.
-
Replace
command
with any of thepbrun
commands such asfq2bam
,bqsr
,applybqsr
, orhaplotypecaller
. -
Please refer to official Parabricks documentation for additional direction.
Earlier Versions
Earlier versions are still available but no longer directly supported by RIS. Please refer to the latest version for direct support.