HPC / LQCD File System Layout

LQCD / HPC users see a file system layout with 5 parts:

/home: Every user will have a home directory after he/she gets an account. Note that for performance and fault tolerance, this is a different home directory from the general lab computing home directory. With a default user quota of 2 GB, /home is not a large file system and is backed up daily. It is designed to store non-data files, such as scripts and executables. The home directory is mounted on interactive nodes and compute nodes. This disk space is managed by individual users.

/work: is an area designed for project or group use. It is organized by project and managed by individual users in the project. This area is NOT backed up. /work/project is designed to store software distributions and small data sets which don't need to be backed up. The software distributions are assumed to be backed up in external source control systems. Data sets are assumed to be easily re-generated or obtained from elsewhere.

/volatile: is a large scratch space to hold files for some moderately long period time. It may be used to hold the output of one job that will later be consumed by a subsequent job. It may also serve as an area to pack and unpack tar balls. In some cases, users work with tens of thousands or hundreds of thousands of small files in one directory, and this is the best place for that type of data. Please don't put these kinds of data on /cache. (Note that if the files need to persist on disk for a long time, /work is a good alternative location.)

/cache: is designed to be a write-through cache front end to the tape library. It is most frequently used to store data files (input and output) for batch jobs. /cache is mounted on the interactive nodes and the compute nodes. This disk area is semi-managed with automatic file migration to tape and automatic removal of files to free up space for later jobs.

/scratch: is a transient storage area available on each compute and interactive node, and is on that node's local disk. This /scratch space on compute nodes will be cleaned up after each job finishes. PLEASE NOTE: Each job node has its own /scratch area. One node cannot see the scratch area of another. In your PBS script when you refer to /scratch you are in fact only referring to the /scratch space for the node which executes the batch script. This directory is not the same as /scratch on the interactive nodes nor on the other compute nodes in a multi-node batch job. Since the /scratch directory on all compute nodes is automatically cleaned by the job epilogue, make sure to save your data from the /scratch space to something more permanent in your job script.

/home and /work are held on the ZFS appliance.  Because it is a single shared resource, bandwidth for a single user can vary widely, from a few 10's of MB/s to a few 100's.   /volatile and /cache are held on Lustre, which has the highest performance of the file systems, especially if striping is enabled, up to ~1 GB/s for a single stream, and slightly more than 9 GB/s aggregate.   /scratch is a local disk, typically limited to ~80 MB/s combined read+write per node.  Since there are over 700 nodes, this still represents the most aggregate bandwidth, > 50 GB/s in all.