Auger Configuration Files

Keywords

All keyword fields are specified as strings. Where a field can contain a list, the list elements should normally be separated by spaces (or new lines), several elements may appear on a single line and the list may continue over several lines. There are only three mandatory keywords, all others are optional and default (where appropriate) to something reasonable. Blank lines are ignored, lines with '#' in column 1 are treated as comments (and ignored).

Note:

1. Use of the INPUT_DATA keyword.

In order to copy a data file called /work/experiment/raw/run001.data to the local disk on the farm specify:

         INPUT_FILES: /work/experiment/raw/run001.data

To copy a data file called /work/experiment/raw/run001.data to the local disk on the farm as a different name called fort.11, specify:

INPUT_FILES: /work/experiment/raw/run001.data
INPUT_DATA: fort.11

This is useful if you want to generate many jobs from a list of input files. One job will be generated for each input file and run on a different farm machine, but all jobs will expect to read data from a file called fort.11 (or whatever you care to call it).

2. Single vs. multiple jobs.

By default the system will generate one job for each data file you specify with the INPUT_FILES keyword. Use the following to change this behaviour:

• SINGLE_JOB - Will force all the input files specified to be processed in one single job. All the specified files will be copied to the farm machine local disk before the job starts executing. Remember that there is a limit of 4 GB disk space per job. It probably does not make sense to specify the INPUT_DATA keyword in this case (see note above).
• MULTI_JOBS - If a single file is (or no files are) specified as INPUT_FILES, run the job this many times - one job per machine. This may be useful for simulations running from the same input file of generated events for example (up to you to make sure that the random number seeds ar different...), or testing.

3. Output files.

The keywords OUTPUT_DATA and OUTPUT_TEMPLATE can be used in pairs to control the disposition of output files. Each OUTPUT_DATA keyword given should have a matching OUTPUT_TEMPLATE. The OUTPUT_DATA is the name of the output file locally on the farm machine disk. The corresponding OUTPUT_TEMPLATE directs the copying of the output file to a mass storage device (the silo or a work area). There are several points to note:

• Using /mss as a prefix to the OUTPUT_TEMPLATE will automatically cause the file to be copied to the tape silo using jput.
• If the template contains the character * then that character will be replaced by the name of the input file being processed.
• If the template contains the string @INPUT_DATA@ will behave in the same way as the * character - the string being replaced by the input file name.
• If the template contains the string @OUTPUT_DATA@, that string will be replaced by the local name of the output file - i.e. the name specified with the keyword OUTPUT_DATA

Auger Job Tracks

All jobs submitted to the farm will belong to a track. The track is used to determine the queue a job runs in and the type of job for accounting purposes.

Special Tracks

Note that there are two special tracks: debug and test. The debug track is useful when you are trying to debug your jobs to make sure that they run correctly. They get the highest priority when running on the farm, but are limited to 4 hours of CPU time. The test track is at the opposite end of the spectrum and is used for jobs that don't have to finish any time soon. Not only do jobs in the test track get the lowest priority for being dispatched to the farm, but even after being dispatched these jobs will be suspended if the farm gets busy.

XML Command File Format

Introduction

This document describes the specification for the XML based job description language (JDL) that Auger uses.

Language Specification

The JDL is XML based. The top level tag will be a request tag. The submission will consist of a description of one or more jobs that are to be run on the farm. The valid subtags and tag attributes will be described below. The script can be augmented with the use of variables. Comments can be specified by using the XML comments by starting the comments with Tags.

There are two tags that are special for describing a job. These are <Request>, and <Job>. The Request tag is the top level tag and is required. Each Job tag corresponds to an individual Farm job. There must be at least one Job inside the Request element. The remainder of the tags specifies properties about the jobs. Note all tags are case sensitive.

• Request - Every submission script must consist of exactly one <Request> tag which is the top level tag. The Request tag must not contain any attributes. All of the properties that describe a submission are specified in subtags. Note that many properties can be specified at either the Request level or the Job level. Properties that are set at the Request level apply to every Job. This allows you to set global properties for each job, rather than having to re-specify them for each individual job

• Job - A job tag corresponds to an individual job for the Farm. This tag takes no attributes, but rather gets all of its values from properties specified with in tags at the Request level, or tag inside this Job tag. There may be any number of these within a request, each one will result in a job on the farm.

• ForEach - The <ForEach> tag is used to create multiple jobs. It acts as a looping mechanism. It has one required attribute, list. The value of this attribute must be the name of a list that was defined using a <List> tag. The ForEach tag is equivalent to having the body of the ForEach repeated once for every element that is in the list, and the element of the list will be stored in a variable with the name of the list. There must be a <Job> tag inside of the ForEach tag (otherwise the ForEach doesnt do anything). ForEach tags can be nested, if desired. In this case, you only have to have a <Job> tag inside the innermost ForEach tag.

• Required Properties - There are certain properties that must be set for a job to be valid. These properties can either be set at the Job level, or at the Request level. Valid tags are <Command>, <Name>, <Project> and <Track>.

Check Description of XML Tags page for more information on the tags.
Check Auger Examples page for detailed examples.