swif2 add-job

Add a new job to an existing workflow.

Usage

swif2 add-job <workflow> [options] [-sbatch <flags> ::] command [args...]

Add a job to workflow wf, specifying that <command [args...]> should be run on the workflow site batch system. You may specify flags to pass directly to the site job submission tool (e.g. sbatch) at launch time with the -sbatch option. The sbatch <flags> arguments will be parsed up to the special terminating string '::' and passed to the batch system at launch time without any validation, though they may be augmented and in some cases overridden by the workflow manager. For example, swif will specify the job name and add flags necessary to support its operation.

Note that, as with commands like ssh, the <command [args...]> that you specify will be parsed by your shell. To prevent special characters from being interpreted while invoking add-job, you must escape them. For example, use single-quotes in bash.

Arguments

Flag Value Comment
-workflow workflow-name Specify the workflow
-account account-name passed to sbatch as -A account-name
-partition partition-name passed to sbatch as -p partition-name
-name job-name Must be unique per-workflow. If ommitted, default is wf-n, where wf is the workflow name and n is the ordinal number of the job.
-antecedent swif-job-name Specify a job that must complete successfully before this job is eligible to run. The named job must already exist within the same workflow. To specify multiple antecedents, repeat the flag for each.
-condition uri Specify a condition that must exist before the job is eligible to run. A condition is a URI that must resolve without error to be considered valid. Supported URI schemes are 'http' and 'file'. To specify multiple conditions, repeat the flag for each. Note that conditions are checked periodically and, once found to be true, will no longer be checked.
-constraint feature[,feature,...] Specify a set of features required by the job.
-cores positive integer How many CPU cores the process will use.
-disk integer[units] Maximum bytes or kilo-, mega-, giga-, terabytes your process will require on the local disk, including inputs and expected scratch usage. You can append any portion of the byte units to the end of the integer, otherwise 'bytes' is assumed. Examples 2g, 1000kilobytes, 3TB. If not specified, swif will request 110% the size of specified inputs. This flag is mutually exclusive with -disk-scratch
-disk-scratch integer[units] Maximum bytes or kilo-, mega-, giga-, terabytes your process will require on the local disk, excluding the size of the inputs (which swif will calculate and add). You can append any portion of the byte units to the end of the integer, otherwise 'bytes' is assumed. Examples 2g, 1000kilobytes, 3TB. This flag is mutually exclusive with -disk
-exclusive Request an exclusive node for the job.
-input local-file remote-uri Process will access the data at remote-uri via local-file. The local file name is always relative to the job's execution directory. To copy a file from the tape library, specify your remote-uri as mss:/mss/.... Or to copy from an accessible filesystem specify file:/.... At present these are the only two schemes supported.
-output local-file remote-uri Like -input, but copying from local-file to remote-uri. You may prefix local-file with match: to specify a shell wildcard pattern that will be evaluated once the job completes to generate a list of matching output files. Note that when using match: remote-uri will be interpreted as a directory. The prefix maybe: indicates that the named file need not exist at successful job completion.
-phase integer Every job may be assigned an integer phase number that indicates what segment of the workflow it belongs to. No jobs belonging to a higher phase will be released to the batch system until all jobs belonging to a lower phase have completed or been abandoned. Once all jobs in a phase have completed, the workflow will advance to the next phase number specified by constituent jobs.PThis can allow graduated introduction of a jobs into the system, or can provide a simple way to implement a form of dependency between stages of a workflow.PIf unspecified, the job is assumed to be runnable at any time.
-ram integer[units] The maximum amount of memory that your process will use. Values specified as described for -disk, above.
-shell interpreter Specify that this job's command should be interpreted by to the given shell (e.g. /bin/bash, /bin/tcsh, /bin/python3, etc)
-site site name If the workflow is associated with multiple sites, you must specify the site for which this job specification is valid. If the workflow is associated with only one site, this flag is not necessary.
-stdout file Redirect standard output to file at compute site. See Output and error files, below.
-stderr file Redirect standard error to file at compute site. See Output and error files, below.
-tag tag-name tag-value You can associate a key/value pair with the job. The tag-name must consist of at most 100 alphanumeric characters, including _, and must start with an alphabetic character. The value can contain regular ascii characters and is limited to 200 characters in length. You can specify multiple tags by using this option more than once.
-time integer[units] Maximum amount of time the job will run. You may append any portion of the units: seconds, minutes, hours, days to the integer value, otherwise seconds is assumed. Example: 5min, 24h, 3days, 1000s.

Output and error files

The file given to -stdout or -stderr must be an absolute path. It may contain two placeholders, which swif fills in each time it launches an attempt of the job:

Without @attempt_id@, every attempt of a job writes to the same file, so a later attempt can overwrite the output that explains why an earlier one failed. Without @job_name@, jobs given the same path write to the same file. Jobs get the same path when a script adds them all with one -stdout, and whenever swif2 modify-jobs changes the path, since it applies one path to every job it selects. Either of these gives every attempt of every job its own file:

-stdout /farm_out/$USER/myrun/@job_name@-@attempt_id@.out
-stdout /farm_out/$USER/myrun/@job_name@/@attempt_id@.out

Placeholders may appear anywhere in the path, including directory names, since swif creates missing directories before it launches the attempt. They must be written exactly as shown: any other @word@, such as @job@ or @Job_Name@, is an error. Only -stdout and -stderr take placeholders; elsewhere, as in -output, swif leaves them as written. Slurm's own filename patterns, such as %j for the Slurm job id, also work, but only in the file name, because swif creates the directories before Slurm expands the patterns.

At jlab/enp, swif always chooses the files. Without -stdout or -stderr it uses /farm_out/$USER/swif/<workflow>/<job>-<attempt>.out and .err. It passes them to sbatch after your -sbatch flags, so swif's choice replaces any --output, -o, --error or -e among those flags. The job's command runs inside a batch script that swif writes, so Slurm never reads #SBATCH lines in your own script there, whatever they set.

At other sites, swif chooses a file only if you give -stdout or -stderr, and then it replaces any -sbatch or #SBATCH setting for that stream. Otherwise Slurm takes it from your -sbatch flags or from #SBATCH lines in the job's script; it reads those lines only when the job has no -shell. Failing both, standard output goes to slurm-<slurm job id>.out in the directory swif creates for the attempt at the site. Standard error goes to the standard output file unless something names a separate one.

swif2 show-job reports the files Slurm says an attempt is using once swif has heard back from Slurm about it. Before that, and for jobs that have not been attempted, it reports the files swif will ask for, which is nothing at sites other than jlab/enp when you gave no -stdout or -stderr. Placeholders in them are filled in, except that @attempt_id@ is left as written for an attempt swif has not created yet, such as a new job's first.