array
The array directive submits tasks as job arrays for executors that support it.
A job array is a collection of jobs with the same resource requirements and the same script (parameterized by an index). Job arrays incur significantly less scheduling overhead compared to individual jobs, and as a result they are preferred by HPC schedulers where possible.
The directive should be specified with a given array size, along with an executor that supports job arrays. For example:
process hello {
executor 'slurm'
array 100
script:
"""
your_command --here
"""
}
Nextflow currently supports job arrays for the following executors:
A process using job arrays collects tasks and submits each batch as a job array when it is ready. Any "leftover" tasks are submitted as a partial job array.
Once a job array is submitted, each "child" task is executed as an independent job. Any tasks that fail (and can be retried) are retried without interfering with the tasks that succeeded. Retried tasks are submitted individually rather than through a job array, in order to allow for the use of dynamic resources.
The following directives must be uniform across all tasks in a process that uses job arrays, because these directives are specified once for the entire job array:
For cloud-based executors like AWS Batch, or when using Fusion with any executor, the following additional directives must be uniform:
When using Wave, the following additional directives must be uniform: