swif2 add-job

Add a new job to an existing workflow.

Usage

swif2 add-job <workflow> [options] [-sbatch <flags> ::] command [args...]

Add a job to workflow wf, specifying that <command [args...]> should be run on the workflow site batch system. You may specify flags to pass directly to the site job submission tool (e.g. sbatch) at launch time with the -sbatch option. The sbatch <flags> arguments will be parsed up to the special terminating string '::' and passed to the batch system at launch time without any validation, though they may be augmented and in some cases overridden by the workflow manager. For example, swif will specify the job name and add flags necessary to support its operation.

Note that, as with commands like ssh, the <command [args...]> that you specify will be parsed by your shell. To prevent special characters from being interpreted while invoking add-job, you must escape them. For example, use single-quotes in bash.

Arguments

Flag	Value	Comment
-workflow	workflow-name	Specify the workflow
-account	account-name	passed to sbatch as -A account-name
-partition	partition-name	passed to sbatch as -p partition-name
-name	job-name	Must be unique per-workflow. If ommitted, default is wf-n, where wf is the workflow name and n is the ordinal number of the job.
-antecedent	swif-job-name	Specify a job that must complete successfully before this job is eligible to run. The named job must already exist within the same workflow. To specify multiple antecedents, repeat the flag for each.
-condition	uri	Specify a condition that must exist before the job is eligible to run. A condition is a URI that must resolve without error to be considered valid. Supported URI schemes are 'http' and 'file'. To specify multiple conditions, repeat the flag for each. Note that conditions are checked periodically and, once found to be true, will no longer be checked.
-constraint	feature[,feature,...]	Specify a set of features required by the job.
-cores	positive integer	How many CPU cores the process will use.
-disk	integer[units]	Maximum bytes or kilo-, mega-, giga-, terabytes your process will require on the local disk, including inputs and expected scratch usage. You can append any portion of the byte units to the end of the integer, otherwise 'bytes' is assumed. Examples 2g, 1000kilobytes, 3TB. If not specified, swif will request 110% the size of specified inputs. This flag is mutually exclusive with -disk-scratch
-disk-scratch	integer[units]	Maximum bytes or kilo-, mega-, giga-, terabytes your process will require on the local disk, excluding the size of the inputs (which swif will calculate and add). You can append any portion of the byte units to the end of the integer, otherwise 'bytes' is assumed. Examples 2g, 1000kilobytes, 3TB. This flag is mutually exclusive with -disk
-exclusive		Request an exclusive node for the job.
-input	local-file remote-uri	Process will access the data at remote-uri via local-file. The local file name is always relative to the job's execution directory. To copy a file from the tape library, specify your remote-uri as `mss:/mss/...`. Or to copy from an accessible filesystem specify `file:/...`. At present these are the only two schemes supported.
-output	local-file remote-uri	Like `-input`, but copying from local-file to remote-uri. You may prefix local-file with `match:` to specify a shell wildcard pattern that will be evaluated once the job completes to generate a list of matching output files. Note that when using `match:` remote-uri will be interpreted as a directory. The prefix `maybe:` indicates that the named file need not exist at successful job completion.
-phase	integer	Every job may be assigned an integer phase number that indicates what segment of the workflow it belongs to. No jobs belonging to a higher phase will be released to the batch system until all jobs belonging to a lower phase have completed or been abandoned. Once all jobs in a phase have completed, the workflow will advance to the next phase number specified by constituent jobs.PThis can allow graduated introduction of a jobs into the system, or can provide a simple way to implement a form of dependency between stages of a workflow.PIf unspecified, the job is assumed to be runnable at any time.
-ram	integer[units]	The maximum amount of memory that your process will use. Values specified as described for -disk, above.
-shell	interpreter	Specify that this job's command should be interpreted by to the given shell (e.g. /bin/bash, /bin/tcsh, /bin/python3, etc)
-site	site name	If the workflow is associated with multiple sites, you must specify the site for which this job specification is valid. If the workflow is associated with only one site, this flag is not necessary.
-stdout	file	Redirect standard output to file at compute site.
-stderr	file	Redirect standard error to file at compute site.
-tag	tag-name tag-value	You can associate a key/value pair with the job. The tag-name must consist of at most 100 alphanumeric characters, including `_`, and must start with an alphabetic character. The value can contain regular ascii characters and is limited to 200 characters in length. You can specify multiple tags by using this option more than once.
-time	integer[units]	Maximum amount of time the job will run. You may append any portion of the units: seconds, minutes, hours, days to the integer value, otherwise seconds is assumed. Example: 5min, 24h, 3days, 1000s.