You are here

Home › Experimental Physics User's Guide

Batch System (Slurm)

Note: Most experimental physics users should be using SWIF2 for optimal interactions with the tape system and to access improved job/workflow management features.

Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. It has been used in many supercomputing sites and data centers across the world. JLab farm deployed Slurm in early 2019, but it was hidden from users by Auger system. Users now can access Slurm directly from farm interactive nodes.

Submitting Batch Jobs:

You can submit jobs from one of the interactive nodes or from within a running batch script. Batch jobs are submitted using slurm sbatch command with a valid project account.  You can specify options on the command line, or (recommended) put all of them into your batch script file. See sample scripts in one of the following sections.  In your batch script, please specify at least the following, plus other options useful to your workflow.

  1. account, using -A, --account=<account>,  Use "sacctmgr show user <username>" or Slurm User Account page to find out which the account <username> belongs to. Send a ServiceNow request if user doesn't have a default slurm account.
  2. partition, which contains a set of nodes and serves like a queue, using -p, --partition=<partition_names>. "sinfo -s" will print out all partition in farm slurm or use Slurm Info page to find out all available slurm partition. Most frequently used partitions are priority and production.  The default and maximum walltime are 4 and 24 hours.  There are a maximum of 32 concurrent priority jobs that can run at one time. There is a limit of 16 queued jobs at any one time per user. So only use priority partition for debug jobs. 
  3. resources needed (number of nodes, mode of nodes, etc), using  -C, --constraint=<list> for set of features of desired nodes, -N, --nodes=<num_node> for multiple node jobs, -n, --ntasks=1 --cpus-per-task=<numcores> for a single node job using multiple cores, and --mem-per-cpu=memsize (in unit of megbytes) for memory in megbytes for each core.
  4. wall time (specifying this more tightly than the default will improve your throughput), using  -t, --time=<time>
  • Note: a computing core denotes a virtual processing core (hyper-threading) not a physical core. A typical physical core contains two virtual processing cores.
 
Job Status and Other Information:
 
Once jobs are submitted into the slurm system, one can use squeue command to check the status of the jobs,  utilize scancel command to cancel one or a list of jobs, and make use of scontrol command to hold jobs. For detailed information about slurm commands, please checkout slurm official documentation.
Powered by Drupal