Auger-Slurm (Beta test)

We are working on a transition of the JLab farm batch system from PBS to Slurm. Starting from Oct. 9, we will allow limited number of users to test this system. During the test period, there are sets of nodes each of which is from a different computing generation, such as 12s, farm14, farm16 and farm18. The hall coordinators have to send the name of test users to sciops@jlab.org so that we can add the users to the Slurm system

How to use this system?

The auger-slurm commands are installed under /site/scicomp/auger-slurm/bin (will be linked to /site/bin in the future) and can be accessed from ifarm or any host where /site is mounted. The jsub and jkill commands have the exactly same interfaces as those in the old auger-pbs,  therefore, the existing jsub scripts will work in the new Auger-Slurm. The slurmHosts replaces the old farmHost and it prints out very different information about the computing nodes. The slurmJobs replaces jobstat in the old pbs system, the notable change is that the output only contains jobs in the slurm system but not includes the jobs are held in the Auger queue, so a newly submitted job will appear in the output of slurmJobs command after auger submit it to slurm when there is less than few hundreds of pending jobs for the user (within few minutes) . The jobinfo is similar to the old jobinfo but outputs more useful information, such as MaxRss, MaxVMSize, AveDiskRead and AvgDiskWrite.

Where is the .out/.err files?

In the auger-slurm system, the default job .out and .err files are no longer copied to the home directory of a user, instead it will be directly written to the central location under /lustre/expphy/farm_out/<user>, name pattern will be JOB_NAME-AUGER_JOB_ID-HOSTNAME.out and JOB_NAME-AUGER_JOB_ID-HOSTNAME.err.  We will have a software to manage this file system, the policy will be announced soon. This change makes the .out and .err file available to user during the run time of a job,  so a user can easily exam the his/her batch job's progress.

How does Slurm establish the envirnment for my job?

Different with PBS, Slurm processes are not run under a shell. The ~/.profile and /.bashrc scripts are not executed as part of the process launch. User have to explicitly prepare their environment before starts the work. Please reference Experimental Nuclear Physics Computing document for the detail information on which file to source.  Please remember to run this command (depends on shell) before call "module load" to load necessary module.

                   source /etc/profile.d/modules.sh        # bash,  sh
                   source /etc/profile.d/modules.csh      # csh, tcsh

How to request the memory?

Different with PBS, Slurm use the real memory to schedule and kill the job, so user can reduce their memory requested in the jsub script. For examples, one user need to ask 60GB for 16 core job in pbs, but only 16GB will be enough for the same job running on slurm.

What OS and NODE_TAG to use?

Users can use slurmHosts client to find out the features of each computing node, the OS and node tags are lists inside the feature. Users can use OS and NODE_TAG to request the correct nodes he/she wants the job to land on.

How to request a exclusive node for the job?

There is a new feature added to Auger-Slurm interface (not available in Auger-PBS). When CPU: 0 or <CPU core="0" /> tag is used, Auger will submit the job to slurm in exclusive mode without request any memory and disk resources. The exclusive job will only land on a empty computing nodes and no other job will schedule to the node.

What are the new features add to Auger-Slurm?

These are the new features added to Auger-Slurm:
  1. Request a while node for a job using "CPU: 0 or <CPU core="0" />
  2. Slurm jsub will take more than one jsub script files when submitting jobs to slurm.
  3. A new optional attribute 'copyOption' (copy or link) is introduced to <Input> xml tag. Reference this Auger document for detail.

Project and Track?

Please continue to use same project and track (the current tracks may be mapped to a different set of queues during the test).