Setting up the Environment

Setting up an ssh key

To access any accelerator, one needs an ssh key to enable an ssh session from a host to an accelerator. There is a lot of information about setting up an ssh key for Linux systems on the internet. However, the following simple steps may be used.

  • Start the SSH-keygen tool by using the following command to generate an RSA authentication key:


[axl@asterisk1 axl]$ ssh-keygen -t rsa

   Generating public/private rsa key pair.
   ...
  • Enter the path to the file that will hold the key: By default, the file name $HOME/.ssh/id_rsa, which represents an RSA v2 key, appears in parentheses.
   Enter file in which to save the key (/home/axl/.ssh/id_rsa): <return>
  • Enter a passphrase for using your key: The passphrase you will enter will be used for encrypting your private key. A good passphrase should be alphanumeric having 10-30 character length. You can also use the null passphrase however it can be a loophole for the security. Note: for this application use a null passphrase so that you can launch batch jobs using ssh without being prompted for a passphrase.
   Enter passphrase (empty for no passphrase): <Type the passphrase>
  • Re-enter the passphrase to confirm it: Type your passphrase once again to confirm it.
   Enter same passphrase again: <Type the passphrase>
   Your identification has been saved in /home/axl/.ssh/id_rsa.
   Your public key has been saved in /home/axl/.ssh/id_rsa.pub.
   The key fingerprint is:
   0b:fa:3c:b8:73:71:bf:58:57:eb:2a:2b:8c:2f:4e:37 axl@myLocalHost
  • Check the Passphrase Key: The private key was saved in .ssh/id_rsa file which is the read-only file. No one else must see the content of that file, as it is used to decrypt all correspondence encrypted with the public key. The public key is saved in .ssh/id_rsa.pub file. The private key file must be read write only by the user who created it.
  • Copy the Public Key to $HOME/.ssh/authorized_keys file: Since the home directory is mounted across all computing clusters including all Phi accelerators, you have to copy the public key onto a $HOME/.ssh/authorized_keys file and make the filesystem permissions 0640, and change the permissions of .ssh to 700 so it is only read/writable by you. Without these permissions, ssh will refuse to use the key. And now you can SSH to the remote systems's account without using a password.

SSH Client Configuration on the computing nodes

In order to simplify the launch of MPI jobs onto accelerators, the ssh configuration settings on the computing nodes have been modified such that users do not need to add host keys of accelerators into $HOME/.ssh/known_hosts file and users do not get any prompt asking permission to add changed host keys of accelerators.

Set up Intel compiler and MPI environment on host machines

Users have to use Intel compilers and Intel MPI to be able to use these accelerators. There may be several Intel compilers installed, but

/opt/intel/composerxe
should point to the most recent version.

Intel MPI is installed in < pre> /opt/intel/impi/4.1.1.036

In order to have correct paths and environment variables, one should add the following lines to your $HOME/.tcshrc ($HOME/.cshrc) if one is using tcsh (csh) on the hosts.

    source /opt/composerxe/bin/compilervars.csh intel64
    source /opt/intel/mkl/bin/mklvars.csh intel64
    source /opt/intel/impi/4.1.1.036/bin/mpivars.csh
    setenv I_MPI_MIC 1

However, if one usees bash (sh) on the hosts, one should add the following lines to ones $HOME/.profile file.

    source /opt/intel/composerxe/bin/compilervars.sh intel64
    source /opt/intel/mkl/bin/mklvars.sh intel64
    source /opt/intel/impi/4.1.1.036/bin/mpivars.sh
    export I_MPI_MIC=1

Additional set up on Phi accelerators

The Linux* environment on the coprocessor utilizes BusyBox to provide a number of Linux* utilities. The usage of these tools may differ slightly when compared to the usage of similar tools provided with the host Linux* distributions.

BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system. For more information about BusyBox, please check out BusyBox Site.

In particular, the Linux environment on the accelerator only offers Bourne Shell (sh) but not bash.

One can launch native (Xeon Phi) jobs on the co-processors directly from the host using MPI, since the /home and /scratch directories are mounted on the devices. If one chooses to use SSH to access the cards, one should ensure that the PATH and LD_LIBRARY_PATHs are set.

 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/opt/intel/impi/4.1.1.036/mic/bin
 LD_LIBRARY_PATH=/opt/intel/impi/4.1.1.036/mic/lib:/opt/intel/composerxe/compiler/lib/mic:/opt/intel/mkl/lib/mic
 I_MPI_MIC=1

Alternatively, when running MPI from the host, one should be able to set

MIC_PATH=/usr/bin:/bin:/usr/sbin:/sbin:/opt/intel/impi/4.1.1.036/mic/bin
MIC_LD_LIBRARY_PATH=/opt/intel/impi/4.1.1.036/mic/lib:/opt/intel/composerxe/compiler/lib/mic:/opt/intel/mkl/lib/mic

If one wants to customize his/her settings for the accelerators, please modify $HOME/.profile file accordingly. Please make sure to use uname -m to check the accelerator machine type in the .profile file to distinguish a host machine from an accelerator node.

On a host:

 uname -m
 x86_64

On an accelerator:

 uname -m
 k1om

Settings of sshd on Phi accelerators

A slightly modified sshd is running on each Phi accelerator so that default LD_LIBRARY_PATH and PATH environment are set up when users are using ssh to launch an executable on the accelerators without login. The reason of running a modified sshd is due to the non-invocation of /etc/profile file by an sshd when a non-interactive ssh session starts.