Currently, the PBS queue for the MIC cluster is testmic. Each computing node is tagged with mic. If one wants to submit a job to the queue, one can do the following:
/usr/local/bin/qsub \ -A xyz \ -c n \ -m ae -M username@jlab.org\ -l nodes=4:mic,walltime=0:10:00\ -N mictest \ -q testmic@qcdpbs \ run_mic.sh
One has to request a multiple of 4 nodes whether one is going to use 4 accelerators or not.
Once a job is scheduled to run on a computing node, the $PBS_NODEFILE contains only host names. We need to change the host names to the corresponding names for the accelerators.
#!/bin/sh rm -f /tmp/jobnodes allnodes=`cat $PBS_NODEFILE | uniq` numnodes=0 for n in $allnodes do echo $n-mic0 >> /tmp/jobnodes echo $n-mic1 >> /tmp/jobnodes echo $n-mic2 >> /tmp/jobnodes echo $n-mic3 >> /tmp/jobnodes numnodes=`expr $numnodes + 4` done cat /tmp/jobnodes echo $numnodes mpiexec.hydra -genvnone -genv I_MPI_FABRICS shm:ofa -genv I_MPI_OFA_ADAPTER_NAME mlx4_0 -genv OMP_NUM_THREADS 174 -gen KMP_AFFINITY scatter -machinefile /tmp/jobnodes -n $numnodes /home/xyzzzz/MIC/t_linalg