/work from within a batch job

Mon, 03/04/2013 - 14:53 — watson

Its recommended that you use the script cache_cp to copy files from /work
into and out of the compute nodes used by your jobs. At this writing,
rcp is symlinked to cache_cp. To use cache_cp, you use it as if you
were using rcp, except that you do not need to provide the correct
server name that is the source or destination under /work. The script will
fill in the appropriate server for you.

Thread Affinity for OpenMP programs and other Multi-core considerations

Mon, 03/04/2013 - 14:33 — watson

The intel compiler provides for several ways of setting thread and core affinity, using the KMP_AFFINITY environment variable:
There are several modes of binding threads, which are described in this article.
However some common options are:

Chroma GPU Example

Mon, 03/04/2013 - 14:13 — watson

GPU versions of Chroma

Pre-compiled versions of Chroma have been ins talled in the /dist/scidac directory.

The root of the directory is


/dist/scidac/chroma_gpu/centos62_cuda50/mvapich2-1.8_omp

Within this directory the installations are:

Using GPUs

Mon, 03/04/2013 - 13:37 — watson

GPUs provide enormous floating point capacity and memory bandwidth, and can yield up to 10x as much performance (science) per dollar, but do require the use of specialized libraries or specialized programming techniques.


Interactive development

The two LQCD / HPC interactive nodes each have one GPU installed which you may use for interactive software development and testing.  All production GPU nodes reached through the gpu queue have 4 GPUs.  

Job Delete & Status

Mon, 03/04/2013 - 13:29 — watson

Delete a Job

Use the PBS command qdel JOBID to delete a job. When the job is hung and qdel JOBID can't remove the job, use qsig -sNULL JOBID to delete the job. Another method to delete a job is qdel -p JOBID.
Unfortunately, a job cannot be deleted if the head node (first node) for the job
is unreachable via the network. This condition is also likely to cause a
flood of emails to the user and/or the admins of the cluster. This is a

Batch system FAQ

Mon, 03/04/2013 - 12:07 — watson

Why isn't my job running, it has been waiting a long time?

There are many possible reasons:

Data Management FAQ

Mon, 03/04/2013 - 11:18 — watson

What is the best/fastest way to move files from /volatile to /cache?

Volatile management

Thu, 02/28/2013 - 16:40 — watson

The /volatile area is managed by a software system that cleans up
the area periodically in multiple steps listed below. The first two
steps will always take place. The third and forth steps will been proceed if the total usage
for all projects of the volatile area is still not below the target.

Typical usage pattern

Thu, 02/28/2013 - 16:34 — watson
  1. A user's workflow script (executed to advance the whole workflow) will call srmGet for files on tape and not currently on disk to stage them to the /cache disk for upcoming batch jobs (calling srmGet for a file already on disk does not cause the file to be re-read)
  2. When the necessary input files are on disk, the batch job is submitted by the workflow script; input files could be srmPin'd to ensure they will stay around until the job launches
  3. When the job starts, it copies the file from /cache to local disk (improves system performance)

Cache manager utilities

Thu, 02/28/2013 - 15:15 — watson

These "storage resource manager" utilities can query status or move files between disk and tape, preserving the files canonical name; i.e. the file has the same name (path) on tape and on disk.

srmProjectInfo projectName
           get project information specified by projectName, information includes project quota, pin quota, available pin quota, and etc.

srmGet path_1 path_2 ...
srmGet -r directory_path
            get file(s) from tape system to cache disk,
            returns: request_id

Pages

Powered by Drupal