Parallel Jobs (MPI)

MPI Libraries:

There are two MPI development and runtime libraries supported on the farm. One is mvapich2 and the other is openmpi. The current version for mvapich is 2.3.2 and 4.0.1 for openmpi. The default path is /usr/local/mvapich2/mvapich2-2.3.2 for mvapich2 and /usr/local/openmpi/openmpi-4.0.1 for openmpi. One can access these two libraries by using module utility on any ifarm and farm nodes.
  • module load openmpi-4.0.1
  • module load mvapich2-2.3.2

Compile MPI applications:

Once one of the above modules is loaded, either /usr/local/mvapich2/mvapich2-2.3.2/bin or /usr/local/openmpi/openmpi-4.0.1 is in the path. Use mpicc, mpic++ to compile C or C++ applications, or use mpifort to compile Fortran applications. The default inter-node communication fabrics for the farm clusters are Infiniband and these two libraries use the network communication. Please use mpicc --show or mpifort --show to see how an application is compiled and linked in detail.

Submit MPI jobs:

On the farm, MPI jobs have to be submitted and run under control of slurm. The direct use of mpirun_rsh from mvapich2 is discouraged, and using mpirun with explicit host names or a machine file is also unnecessary. The following two sample sbatch scripts show how to submit a MPI job to the farm.

mvapich2:

#!/bin/bash -l
#SBATCH -A youraccount
#SBATCH -p production (or general)
#SBATCH -N numnodes
#SBATCH --exclusive
#SBATCH -t hour:min:seconds
#SBATCH -J jobname
#SBATCH -C special features (like farm18)

srun --mpi=pmi2 -n numnodes yourexec arg0 arg1


openmpi:

#!/bin/bash -l
#SBATCH -A youraccount
#SBATCH -p production (or general)
#SBATCH -N numnodes
#SBATCH --exclusive
#SBATCH -t hour:min:seconds
#SBATCH -J jobname
#SBATCH -C special features (like farm18)

/usr/local/openmpi/openmpi-4.0.1/bin/mpirun --mca btl_openib_allow_ib 1 yourexec arg0 arg1