Running

We are encouraging users to compile their applications to run natively on Phi accelerators. The offload mode can be easily achieved by using compiler directives. To read more about offloading mode, check out Native and Offload Programming Models

On the login node

Once an application is built for Phi accelerators, you can ssh into the mic node (qcd12kmi-mic0) and run the application. By doing this way, you can check whether the application is operating correctly and efficiently.

Run MPI programs

All MPI programs should be launched using mpiexec.hydra. To find out the options of this command, one can type:

   mpiexec.hydra --help

or read detailed documents at /opt/intel/impi/4.1.0/doc.

One can test an MPI program on the login node by testing communications between the host and the accelerator. This means one has to compile two versions of the application: one for the host and one for the accelerator. Let us assume the host application is mpitest.host and the mic application is mpitest.mic. The following command shows how to run MPI programs between a host and an accelerator:

 mpiexec.hydra -genv I_MPI_FABRICS shm:ofa -host qcd12kmi-mic0 -env I_MPI_OFA_ADAPTER_NAME mlx4_0 -n 1 mpitest.mic : 
 -host qcd12kmi -env I_MPI_OFA_ADAPTER_NAME mlx4_0 -env LD_LIBRARY_PATH 
 /opt/intel/impi/4.1.0.024/mic/lib:/opt/intel/composerxe/compiler/lib/mic:/opt/intel/mkl/lib/mic -n 1 mpitest.mic


One can launch an MPI program to run on multiple accelerators using 2 different ways. Since the LD_LIBRARY_PATH and PATH environment variables on an accelerators are different from these on a host, one needs to tell MPI launcher these environment variables.

1) Setting LD_LIBRARY_PATH explicitly in an MPI launch command. For example:

 mpiexec.hydra -genv I_MPI_FABRICS shm:ofa -genv I_MPI_OFA_ADAPTER_NAME mlx4_0 -genv LD_LIBRARY_PATH 
 /opt/intel/impi/4.1.0.024/mic/lib:/opt/intel/composerxe/compiler/lib/mic:/opt/intel/mkl/lib/mic -machinefile nodes -n 4 mpitest.mic


2) Let the modified sshd use the default environment variables on accelerators. However, mpiexec.hydra will pass all environment variables on a host when the mpiexec.hydra is invoked. Most likely, the LD_LIBRARY_PATH for a host is wrong for an accelerator. Therefore, one has to tell mpiexec.hydra not to pass host environment variables.

 mpiexec.hydra -genvnone -genv I_MPI_FABRICS shm:ofa -genv I_MPI_OFA_ADAPTER_NAME mlx4_0 -machinefile nodes -n 4 mpitest.mic