GPU Jobs

GPUs are available on the batch farm for both interactive and batch processing using slurm commands. GPU use from Auger is not supported. The slurm commands (sbatch, salloc, etc) are available on the ifarm machines.


The following GPU resources are available:


·      TitanRTX (up to four per node)



Batch use

Submissions should request one to four GPUs, and should use the GPU partition. The environment variable CUDA_VISIBLE_DEVICES will be set to show which GPUs are allocated for your job. Here is an example script that allocates two Nvidia Titan RTX GPU on a single node and shows the execution environment in the slurm output file.



#slurm sbatch requesting two K20 GPUs. CUDA_VISIBLE_DEVICES will be set.

#SBATCH --nodes 1

#SBATCH --partition gpu

#SBATCH --cpus-per-task 1

#SBATCH --gres=gpu:TitanRTX:2

#SBATCH --job-name=GPU-test


echo =================


echo =================



Interactive Use


For interactive use of GPUS, slurm is used to allocate GPU resources and avoid conflicts. Here is an example of starting an interactive session on a GPU machine with a single GPU of any sort:


$ salloc -n 1 -p gpu --gres=gpu:TitanRTX":1

salloc: Granted job allocation 4474492

$ srun --pty bash

bash-4.2$ hostname


bash-4.2$ echo $CUDA_VISIBLE_DEVICES


$ exit


salloc: Relinquishing job allocation 4475870