matflow.SlurmPosix

class matflow.SlurmPosix(directives=None, options=None, submit_cmd=None, show_cmd=None, del_cmd=None, js_cmd=None, login_nodes_cmd=None, array_switch=None, array_item_var=None, *args, **kwargs)

Bases: SlurmPosix

A scheduler that uses SLURM.

Keyword Arguments:

directives (dict) – Scheduler directives. Each item is written verbatim in the jobscript as a scheduler directive, and is not processed in any way. If a value is None, the key is considered a flag-like directive. If a value is a list, multiple directives will be printed to the jobscript with the same key, but different values.

Parameters:

• directives (dict | None) –

• options (dict | None) –

• submit_cmd (str | None) –

• show_cmd (Sequence[str] | None) –

• del_cmd (str | None) –

• js_cmd (str | None) –

• login_nodes_cmd (Sequence[str] | None) –

• array_switch (str | None) –

• array_item_var (str | None) –

Notes

• runs in current working directory by default [2]

References

[1] https://manpages.org/sbatch [2] https://ri.itservices.manchester.ac.uk/csf4/batch/sge-to-slurm/

Methods

cancel_jobs	Cancel submitted jobs.
format_directives	Format the directives to the scheduler.
format_switch	Format a particular switch to use the JS command.
get_job_state_info	Query the scheduler to get the states of all of this user's jobs, optionally filtering by specified job IDs.
get_std_out_err_filename	File name of combined standard output and error streams.
get_stderr_filename	File name of the standard error stream file.
get_stdout_filename	File name of the standard output stream file.
get_submit_command	Get the command to use to submit a job to the scheduler.
get_version_info	Get the version of the scheduler.
is_jobscript_active	Query if a jobscript is running/pending.
is_num_cores_supported	Test whether particular number of cores is supported in given range of cores.
parse_submission_output	Extract scheduler reference for a newly submitted jobscript
process_resources	Perform scheduler-specific processing to the element resources.
wait_for_jobscripts	Wait for jobscripts to update their state.

Attributes

DEFAULT_ARRAY_ITEM_VAR	Default shell variable with array ID.
DEFAULT_ARRAY_SWITCH	Default prefix to enable array processing.
DEFAULT_DEL_CMD	Default cancel command.
DEFAULT_JS_CMD	Default job control directive prefix.
DEFAULT_LOGIN_NODES_CMD	Default command for logging into nodes.
DEFAULT_LOGIN_NODE_MATCH	Default pattern for matching the names of login nodes.
DEFAULT_SHOW_CMD	Default command to show the queue state.
DEFAULT_SUBMIT_CMD	Default submission command.
INTER_STATE_QUERY_DELAY	Delay (in seconds) between attempts to query the state.
NUM_STATE_QUERY_TRIES	Number of times to try when querying the state.
state_lookup	Maps scheduler state codes to JobscriptElementState values.
unique_properties	Unique properties, for hashing.

DEFAULT_ARRAY_ITEM_VAR: ClassVar[str] = 'SLURM_ARRAY_TASK_ID'

Default shell variable with array ID.

DEFAULT_ARRAY_SWITCH: ClassVar[str] = '--array'

Default prefix to enable array processing.

DEFAULT_DEL_CMD: ClassVar[str] = 'scancel'

Default cancel command.

DEFAULT_JS_CMD: ClassVar[str] = '#SBATCH'

Default job control directive prefix.

DEFAULT_LOGIN_NODES_CMD: ClassVar[Sequence[str] | None] = None

Default command for logging into nodes.

DEFAULT_LOGIN_NODE_MATCH: ClassVar[str] = '*login*'

Default pattern for matching the names of login nodes.

DEFAULT_SHOW_CMD: ClassVar[Sequence[str]] = ('squeue', '--me')

Default command to show the queue state.

DEFAULT_SUBMIT_CMD: ClassVar[str] = 'sbatch'

Default submission command.

INTER_STATE_QUERY_DELAY: ClassVar[float] = 0.5

Delay (in seconds) between attempts to query the state.

NUM_STATE_QUERY_TRIES: ClassVar[int] = 5

Number of times to try when querying the state.

cancel_jobs(js_refs, jobscripts=None)

Cancel submitted jobs.

Parameters:

• js_refs (list[str]) –

• jobscripts (list[Jobscript] | None) –

format_directives(resources, num_elements, is_array, sub_idx, js_idx)

Format the directives to the scheduler.

Parameters:

• resources (ElementResources) –

• num_elements (int) –

• is_array (bool) –

• sub_idx (int) –

• js_idx (int) –

Return type:

str

format_switch(switch)

Format a particular switch to use the JS command.

Parameters:

switch (str) –

Return type:

str

get_job_state_info(*, js_refs=None)

Query the scheduler to get the states of all of this user’s jobs, optionally filtering by specified job IDs.

Jobs that are not in the scheduler’s status output will not appear in the output of this method.

Parameters:

js_refs (Sequence[str] | None) –

Return type:

Mapping[str, JobscriptElementState | Mapping[int, JobscriptElementState]]

get_std_out_err_filename(js_idx, job_ID, array_idx=None)

File name of combined standard output and error streams.

Notes

We use the standard output stream filename format for the combined output and error streams file.

Parameters:

• js_idx (int) –

• job_ID (str) –

• array_idx (int | None) –

get_stderr_filename(js_idx, job_ID, array_idx=None)

File name of the standard error stream file.

Parameters:

• js_idx (int) –

• job_ID (str) –

• array_idx (int | None) –

Return type:

str

get_stdout_filename(js_idx, job_ID, array_idx=None)

File name of the standard output stream file.

Parameters:

• js_idx (int) –

• job_ID (str) –

• array_idx (int | None) –

Return type:

str

get_submit_command(shell, js_path, deps)

Get the command to use to submit a job to the scheduler.

Return type:

List of argument words.

Parameters:

• shell (Shell) –

• js_path (str) –

• deps (dict[Any, tuple[Any, ...]]) –

get_version_info()

Get the version of the scheduler.

Return type:

VersionInfo

is_jobscript_active(job_ID)

Query if a jobscript is running/pending.

Parameters:

job_ID (str) –

Return type:

bool

static is_num_cores_supported(num_cores, core_range)

Test whether particular number of cores is supported in given range of cores.

Parameters:

• num_cores (int | None) –

• core_range (Sequence[int]) –

Return type:

bool

parse_submission_output(stdout)

Extract scheduler reference for a newly submitted jobscript

Parameters:

stdout (str) –

Return type:

str

classmethod process_resources(resources, scheduler_config)

Perform scheduler-specific processing to the element resources.

Note

This mutates resources.

Parameters:

• resources (ElementResources) –

• scheduler_config (SchedulerConfigDescriptor) –

Return type:

None

state_lookup: ClassVar[Mapping[str, JobscriptElementState]] = {'CANCELLED': JobscriptElementState.cancelled, 'COMPLETED': JobscriptElementState.finished, 'COMPLETING': JobscriptElementState.running, 'FAILED': JobscriptElementState.errored, 'OUT_OF_MEMORY': JobscriptElementState.errored, 'PENDING': JobscriptElementState.pending, 'RUNNING': JobscriptElementState.running, 'TIMEOUT': JobscriptElementState.errored}

Maps scheduler state codes to JobscriptElementState values.

property unique_properties: tuple[str, str, Any, Any]

Unique properties, for hashing.

wait_for_jobscripts(js_refs)

Wait for jobscripts to update their state.

Parameters:

js_refs (list[str]) –

Return type:

None