matflow.Submission#

class matflow.Submission(index, jobscripts, workflow=None, at_submit_metadata=None, JS_parallelism=None, environments=None)#

Bases: Submission

A collection of jobscripts to be submitted to a scheduler.

Parameters:

index (int) – The index of this submission.
jobscripts (list[Jobscript]) – The jobscripts in the submission.
workflow (Workflow) – The workflow this is part of.
submission_parts (dict) – Description of submission parts.
JS_parallelism (bool) – Whether to exploit jobscript parallelism.
environments (EnvironmentsList) – The execution environments to use.
at_submit_metadata (dict[str, Any] | None) –

Methods

`cancel`	Cancel the active jobs for this submission's jobscripts.
`from_json_like`	Make an instance of this class from JSON (or YAML) data.
`get_active_jobscripts`	Get jobscripts that are active on this machine, and their active states.
`get_app_log_file_name`	The app log file name.
`get_app_log_file_path`	The file path to the app log, for the specified submission.
`get_app_log_path`	The path to the app log directory for this submission, for the specified submission.
`get_app_std_path`	The path to the app standard output and error stream files directory, for the specified submission.
`get_commands_path`	The path to the directory containing command files, for the specified submission.
`get_end_time`	Get the end time of a given submission part.
`get_jobscript_functions_name`	Get the name of the jobscript functions file for the specified shell.
`get_jobscript_functions_path`	Get the path of the jobscript functions file for the specified shell.
`get_js_funcs_path`	The path to the directory containing the shell functions that are invoked within jobscripts and commmand files, for the specified submission.
`get_js_path`	The path to the jobscript files directory, for the specified submission.
`get_js_run_ids_path`	The path to the directory containing jobscript run IDs, for the specified submission.
`get_js_script_indices_path`	The path to the directory containing script indices for combined-script jobscripts only, for the specified submission.
`get_js_std_path`	The path to the jobscript standard output and error files directory, for the specified submission.
`get_js_win_pids_path`	The path to the directory containing process ID files (Windows only), for the specified submission.
`get_path`	The directory path to files associated with the specified submission.
`get_process_IDs`	Return jobscript process IDs.
`get_scheduler_job_IDs`	Return jobscript scheduler job IDs.
`get_scripts_path`	The path to the directory containing action scripts, for the specified submission.
`get_start_time`	Get the start time of a given submission part.
`get_tmp_path`	The path to the temporary files directory, for the specified submission.
`get_unique_schedulers`	Get unique schedulers and which of this submission's jobscripts they correspond to.
`get_unique_schedulers_of_jobscripts`	Get unique schedulers and which of the passed jobscripts they correspond to.
`get_unique_shells`	Get unique shells and which jobscripts they correspond to.
`list_jobscripts`	Print a table listing jobscripts and associated information.
`list_task_jobscripts`	Print a table listing the jobscripts associated with the specified (or all) tasks for the specified submission.
`submit`	Generate and submit the jobscripts of this submission.
`to_dict`	Serialize this object as a dictionary.
`to_json_like`	Serialize this object as an object structure that can be trivially converted to JSON.

Attributes

`APP_STD_DIR_NAME`
`COMMANDS_DIR_NAME`
`EARs_by_elements`	All EARs in this submission, grouped by element.
`JS_DIR_NAME`
`JS_FUNCS_DIR_NAME`
`JS_RUN_IDS_DIR_NAME`
`JS_SCRIPT_INDICES_DIR_NAME`
`JS_STD_DIR_NAME`
`JS_WIN_PIDS_DIR_NAME`
`JS_parallelism`	Whether to exploit jobscript parallelism.
`LOG_DIR_NAME`
`SCRIPTS_DIR_NAME`
`TMP_DIR_NAME`
`WORKFLOW_APP_ALIAS`
`all_EAR_IDs`	The IDs of all EARs in this submission.
`all_EARs`	All EARs in this submission.
`all_EARs_IDs_by_jobscript`
`all_EARs_by_jobscript`
`app_log_path`	The path to the app log directory for this submission for this submission.
`app_std_path`	The path to the app standard output and error stream files directory, for the this submission.
`at_submit_metadata`
`commands_path`	The path to the directory containing command files, for this submission.
`end_time`	Get the final non-None end time over all submission parts.
`environments`	The execution environments to use.
`index`	The index of this submission.
`is_scheduled`	Return whether each jobscript of this submission uses a scheduler or not.
`jobscript_indices`	All associated jobscript indices.
`jobscripts`	The jobscripts in this submission.
`js_funcs_path`	The path to the directory containing the shell functions that are invoked within jobscripts and commmand files, for this submission.
`js_path`	The path to the jobscript files directory, for this submission.
`js_run_ids_path`	The path to the directory containing jobscript run IDs, for this submission.
`js_script_indices_path`	The path to the directory containing script indices for combined-script jobscripts only, for this submission.
`js_std_path`	The path to the jobscript standard output and error files directory, for this submission.
`js_win_pids_path`	The path to the directory containing process ID files (Windows only), for this submission.
`needs_app_log_dir`	Whether this submision requires an app log directory.
`needs_script_indices_dir`	Whether this submision requires a directory for combined-script script ID files.
`needs_submit`	Whether this submission needs a submit to be done.
`needs_win_pids_dir`	Whether this submision requires a directory for process ID files (Windows only).
`outstanding_jobscripts`	Jobscript indices that have not yet been successfully submitted.
`path`	The path to the directory containing action scripts.
`scripts_path`	The path to the directory containing action scripts, for this submission.
`start_time`	Get the first non-None start time over all submission parts.
`status`	The status of this submission.
`submission_parts`
`submitted_jobscripts`	Jobscript indices that have been successfully submitted.
`tmp_path`	The path to the temporary files directory for this submission.
`workflow`	The workflow this is part of.

APP_STD_DIR_NAME = 'app_std'#

COMMANDS_DIR_NAME = 'commands'#

property EARs_by_elements: Mapping[int, Mapping[int, Sequence[ElementActionRun]]]#: All EARs in this submission, grouped by element.

JS_DIR_NAME = 'jobscripts'#

JS_FUNCS_DIR_NAME = 'js_funcs'#

JS_RUN_IDS_DIR_NAME = 'js_run_ids'#

JS_SCRIPT_INDICES_DIR_NAME = 'js_script_indices'#

JS_STD_DIR_NAME = 'js_std'#

JS_WIN_PIDS_DIR_NAME = 'js_pids'#

property JS_parallelism: bool | Literal['direct', 'scheduled'] | None#: Whether to exploit jobscript parallelism.

LOG_DIR_NAME = 'app_logs'#

SCRIPTS_DIR_NAME = 'scripts'#

TMP_DIR_NAME = 'tmp'#

WORKFLOW_APP_ALIAS = 'wkflow_app'#

property all_EAR_IDs: Iterable[int]#: The IDs of all EARs in this submission.

property all_EARs: Iterable[ElementActionRun]#: All EARs in this submission.

property all_EARs_IDs_by_jobscript: list[numpy.ndarray]#

property all_EARs_by_jobscript: list[list[ElementActionRun]]#

property app_log_path: Path#: The path to the app log directory for this submission for this submission.

property app_std_path: Path#: The path to the app standard output and error stream files directory, for the this submission.

property at_submit_metadata: dict[str, dict[str, Any]]#

cancel()#

Cancel the active jobs for this submission’s jobscripts.

Return type:: None

property commands_path: Path#: The path to the directory containing command files, for this submission.

property end_time: datetime | None#: Get the final non-None end time over all submission parts.

property environments: EnvironmentsList#: The execution environments to use.

classmethod from_json_like(json_like, shared_data=None)#

Make an instance of this class from JSON (or YAML) data.

Parameters:

json_like (str | Mapping[str, JSONed] | Sequence[Mapping[str, JSONed]] | None) – The data to deserialise.
shared_data (Mapping[str, ObjectList[JSONable]] | None) – Shared context data.

Return type:

The deserialised object.

get_active_jobscripts(as_json=False)#

Get jobscripts that are active on this machine, and their active states.

Parameters:: as_json (Literal[True] | Literal[False]) –
Return type:: Mapping[int, Mapping[int, Mapping[int, JobscriptElementState | str]]]

static get_app_log_file_name(run_ID)#

The app log file name.

Parameters:: run_ID (int | str) –
Return type:: str

classmethod get_app_log_file_path(submissions_path, sub_idx, run_ID)#

The file path to the app log, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –
run_ID (int) –

classmethod get_app_log_path(submissions_path, sub_idx)#

The path to the app log directory for this submission, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_app_std_path(submissions_path, sub_idx)#

The path to the app standard output and error stream files directory, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_commands_path(submissions_path, sub_idx)#

The path to the directory containing command files, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

get_end_time(submit_time)#

Get the end time of a given submission part.

Parameters:: submit_time (str) –
Return type:: datetime | None

get_jobscript_functions_name(shell, shell_idx)#

Get the name of the jobscript functions file for the specified shell.

Parameters:

shell (Shell) –
shell_idx (int) –

Return type:

str

get_jobscript_functions_path(shell, shell_idx)#

Get the path of the jobscript functions file for the specified shell.

Parameters:

shell (Shell) –
shell_idx (int) –

Return type:

Path

classmethod get_js_funcs_path(submissions_path, sub_idx)#

The path to the directory containing the shell functions that are invoked within jobscripts and commmand files, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_js_path(submissions_path, sub_idx)#

The path to the jobscript files directory, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_js_run_ids_path(submissions_path, sub_idx)#

The path to the directory containing jobscript run IDs, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_js_script_indices_path(submissions_path, sub_idx)#

The path to the directory containing script indices for combined-script jobscripts only, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_js_std_path(submissions_path, sub_idx)#

The path to the jobscript standard output and error files directory, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_js_win_pids_path(submissions_path, sub_idx)#

The path to the directory containing process ID files (Windows only), for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

classmethod get_path(submissions_path, sub_idx)#

The directory path to files associated with the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

get_process_IDs()#

Return jobscript process IDs.

Return type:: tuple[int, …]

get_scheduler_job_IDs()#

Return jobscript scheduler job IDs.

Return type:: tuple[str, …]

classmethod get_scripts_path(submissions_path, sub_idx)#

The path to the directory containing action scripts, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

get_start_time(submit_time)#

Get the start time of a given submission part.

Parameters:: submit_time (str) –
Return type:: datetime | None

classmethod get_tmp_path(submissions_path, sub_idx)#

The path to the temporary files directory, for the specified submission.

Parameters:

submissions_path (Path) –
sub_idx (int) –

Return type:

Path

get_unique_schedulers()#

Get unique schedulers and which of this submission’s jobscripts they correspond to.

Returns:: Mapping where keys are a sequence of jobscript index descriptors and the values are the scheduler to use for that jobscript. A jobscript index descriptor is a pair of the submission index and the main jobscript index.
Return type:: scheduler_mapping

static get_unique_schedulers_of_jobscripts(jobscripts)#

Get unique schedulers and which of the passed jobscripts they correspond to.

Uniqueness is determined only by the QueuedScheduler.unique_properties tuple.

Parameters:: jobscripts (list[Jobscript]) –
Returns:: Mapping where keys are a sequence of jobscript index descriptors and the values are the scheduler to use for that jobscript. A jobscript index descriptor is a pair of the submission index and the main jobscript index.
Return type:: scheduler_mapping

get_unique_shells()#

Get unique shells and which jobscripts they correspond to.

Return type:: Iterable[tuple[tuple[int, …], Shell]]

property index: int#: The index of this submission.

property is_scheduled: tuple[bool, ...]#: Return whether each jobscript of this submission uses a scheduler or not.

property jobscript_indices: tuple[int, ...]#: All associated jobscript indices.

property jobscripts: list[Jobscript]#: The jobscripts in this submission.

property js_funcs_path: Path#: The path to the directory containing the shell functions that are invoked within jobscripts and commmand files, for this submission.

property js_path: Path#: The path to the jobscript files directory, for this submission.

property js_run_ids_path: Path#: The path to the directory containing jobscript run IDs, for this submission.

property js_script_indices_path: Path#: The path to the directory containing script indices for combined-script jobscripts only, for this submission.

property js_std_path: Path#: The path to the jobscript standard output and error files directory, for this submission.

property js_win_pids_path: Path#: The path to the directory containing process ID files (Windows only), for this submission.

list_jobscripts(max_js=None, jobscripts=None, width=None)#

Print a table listing jobscripts and associated information.

Parameters:

max_js (int | None) – Maximum jobscript index to display. This cannot be specified with jobscripts.
jobscripts (list[int] | None) – A list of jobscripts to display. This cannot be specified with max_js.
width (int | None) – Width in characters of the printed table.

Return type:

None

list_task_jobscripts(task_names=None, max_js=None, width=None)#

Print a table listing the jobscripts associated with the specified (or all) tasks for the specified submission.

Parameters:

task_names (list[str] | None) – List of sub-strings to match to task names. Only matching task names will be included.
max_js (int | None) – Maximum jobscript index to display.
width (int | None) – Width in characters of the printed table.

Return type:

None

property needs_app_log_dir: bool#: Whether this submision requires an app log directory.

property needs_script_indices_dir: bool#: Whether this submision requires a directory for combined-script script ID files.

property needs_submit: bool#: Whether this submission needs a submit to be done.

property needs_win_pids_dir: bool#: Whether this submision requires a directory for process ID files (Windows only).

property outstanding_jobscripts: tuple[int, ...]#: Jobscript indices that have not yet been successfully submitted.

property path: Path#: The path to the directory containing action scripts.

property scripts_path: Path#: The path to the directory containing action scripts, for this submission.

property start_time: datetime | None#: Get the first non-None start time over all submission parts.

property status: SubmissionStatus#: The status of this submission.

property submission_parts: list[SubmissionPart]#

submit(status, ignore_errors=False, print_stdout=False, add_to_known=True)#

Generate and submit the jobscripts of this submission.

Parameters:

status (Status | None) –
ignore_errors (bool) –
print_stdout (bool) –
add_to_known (bool) –

Return type:

list[int]

property submitted_jobscripts: tuple[int, ...]#: Jobscript indices that have been successfully submitted.

property tmp_path: Path#: The path to the temporary files directory for this submission.

to_dict()#

Serialize this object as a dictionary.

Return type:: dict[str, Any]

to_json_like(dct=None, shared_data=None, exclude=(), path=None)#

Serialize this object as an object structure that can be trivially converted to JSON. Note that YAML can also be produced from the result of this method; it just requires a different final serialization step.

Parameters:

dct (dict[str, JSONable] | None) –
shared_data (_JSONDeserState) –
exclude (Container[str | None]) –
path (list | None) –

Return type:

tuple[JSONDocument, _JSONDeserState]

property workflow: Workflow#: The workflow this is part of.