JLab Scientific Computing

On Computing nodes using PBS

Currently, the PBS queue for the MIC cluster is testmic. Each computing node is tagged with mic. If one wants to submit a job to the queue, one can do the following:

   /usr/local/bin/qsub \
   -A xyz \
   -c n \
   -m ae -M username@jlab.org\
   -l nodes=4:mic,walltime=0:10:00\
   -N mictest \
   -q testmic@qcdpbs \
   run_mic.sh

One has to request a multiple of 4 nodes whether one is going to use 4 accelerators or not.

Example MPI script for a PBS job

Once a job is scheduled to run on a computing node, the $PBS_NODEFILE contains only host names. We need to change the host names to the corresponding names for the accelerators.

    #!/bin/sh
    rm -f /tmp/jobnodes
    allnodes=`cat $PBS_NODEFILE | uniq`

    numnodes=0
    for n in $allnodes
    do
       echo $n-mic0 >> /tmp/jobnodes
       echo $n-mic1 >> /tmp/jobnodes
       echo $n-mic2 >> /tmp/jobnodes
       echo $n-mic3 >> /tmp/jobnodes
       numnodes=`expr $numnodes + 4` 
    done
    
    cat /tmp/jobnodes

    echo $numnodes

    mpiexec.hydra -genvnone -genv I_MPI_FABRICS shm:ofa -genv I_MPI_OFA_ADAPTER_NAME mlx4_0 -genv OMP_NUM_THREADS 174 
    -gen KMP_AFFINITY scatter -machinefile /tmp/jobnodes -n $numnodes /home/xyzzzz/MIC/t_linalg