Page Comparison

...

Using Ansys/Fluent in OnDemand

Open an OnDemand Cluster Desktop from https://compute.engr.wustl.edu

Open a terminal, and execute the commands:

/project/research/ansys21/fluent21

Multi-node computing with the OnDemand GUI

Start a VNC job as normal. Once there, you can reserve original Infiniband nodes with:

/project/research/ansys/reserve_fluent.sh X

where X is the number of cores you wish to reserve.

You may reserve nodes within the CPU Compute queue with:

/project/research/ansys/reserve_fluent_cpu.sh X
/project/research/ansys/reserve_fluent_cpu_long.sh X

for either the 7 day or long 21 day queues. Keep in mind this reservation job will end independently of any other Ansys application utilizing it as a target for jobs.

The script will output information you need to continue:

Starting a 4 node Fluent Reservation
Reserving a 4 IB CPU Fluent Job
...starting job:

Your job is:
1234567 seasuser   PEND  ib1        ssh.engr.wustl.edu    -        Fluent-MPI-Waiter Nov  1 11:11

Please look for a file in the root of your home directory :
fluenthosts.1234567
(where 1234567 is the job number you are given above)
and pass that to Fluent as:
fluent -cnf=/home/research/username/fluenthosts.1234567
  (where username is your own WUSTL Key)

You must remember to kill this job when you are done with Fluent!
Use the command:
bkill 1234567
The nodes you reserve will not be released until you do.

Note the job number and the “fluenthosts.1234567” filename - your number will be different for your own started job.

Run Fluent in the VNC session as this example, with the above information, modifying for your own needs:

/project/research/ansys21/fluent21 3ddp -tX -cnf=/home/research/username/fluenthosts.1234567 -pib.ofed -gui_machine=$HOSTNAME -i test.jou

For the normal queue via Ethernet, change

-pib.ofed

to

-peth.ofed


The ANSYS GUI program can run in the queues dedicated to VNC jobs and control jobs within the cpu-compute-* queues without issue. 

Using Ansys/Fluent in Batch Mode

A sample batch job for running Fluent is below, geared towards running an Infiniband-based MPI job on the cpu-compute queue. It expects to run from a current working directory of “/storage1/piname/Active/project/”, which should be modified everywhere in the script for your specific requirements.

It asks for 64 total CPUs, tiling them in groups of 8 across the available nodes, and 32GB of RAM per node.

#BSUB -R '(!gpu)'
#BSUB -n 64
#BSUB -o /storage1/piname/Active/project/ansys.out
#BSUB -J fluentJob
#BSUB -R "rusage[mem=32]"
#BSUB -R "span[ptile=8]"
#BSUB -q cpu-compute

export LSF_ENABLED=1
cd $LS_SUBCWD
FL_SCHEDULER_HOST_FILE=lsf.${LSB_JOBID}.hosts
/bin/rm -rf ${FL_SCHEDULER_HOST_FILE}
if [ -n "$LSB_MCPU_HOSTS" ]; then
    HOST=""
    COUNT=0
    for i in $LSB_MCPU_HOSTS
    do
      if [ -z "$HOST" ]; then
         HOST="$i"
      else
         echo "$HOST:$i" >> $FL_SCHEDULER_HOST_FILE
         COUNT=`expr $COUNT + $i`
         HOST=""
      fi
    done
fi

/project/research/ansys21/fluent21 2ddp -g -t64 -scheduler_tight_coupling -pib.ofed -i/storage1/piname/Active/project/testJournal.jou -pcheck -setenv=FLUENT_ARCH=lnamd64 -alnamd64 -env -setenv=LD_LIBRARY_PATH=/project/research/ansys21/gcc/lib2:/project/research/ansys21/gcc/lib2:/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/lib -setenv=FLUENT_ARCH=lnamd64 -cnf=${FL_SCHEDULER_HOST_FILE}

rm lsf.${LSB_JOBID}.hosts

---

The various BSUB parameters are explained earlier in this documentation.

The first part of this script aids in capturing the hosts the job is destined to run on, and is required.

The second covers starting Fluent in 2ddp mode; indicating the number of tasks, scheduler options, Ethernet MPI, input file, parallel check function, and then setting various environmental variables and architecture settings.

The last line cleans up the MPI hosts file.