matflow.Jobscript

class matflow.Jobscript(task_insert_IDs, task_actions, task_elements, EAR_ID, resources, task_loop_idx, dependencies, submit_time=None, submit_hostname=None, submit_machine=None, submit_cmdline=None, scheduler_job_ID=None, process_ID=None, version_info=None, os_name=None, shell_name=None, scheduler_name=None, running=None)

Bases: Jobscript

A group of actions that are submitted together to be executed by the underlying job management system as a single unit.

Parameters:

• task_insert_IDs (list[int]) – The task insertion IDs.

• task_actions (list[tuple]) – The actions of the tasks. task insert ID, action_idx, index into task_loop_idx for each JS_ACTION_IDX

• task_elements (dict[int, list[int]]) – The elements of the tasks. Maps JS_ELEMENT_IDX to list of TASK_ELEMENT_IDX for each TASK_INSERT_ID

• EAR_ID (NDArray) – Element action run information.

• resources (ElementResources) – Resources to use

• task_loop_idx (list[dict]) – Description of what loops are in play.

• dependencies (dict[int, dict]) – Description of dependencies.

• submit_time (datetime) – When the jobscript was submitted, if known.

• submit_hostname (str) – Where the jobscript was submitted, if known.

• submit_machine (str) – Description of what the jobscript was submitted to, if known.

• submit_cmdline (str) – The command line used to do the commit, if known.

• scheduler_job_ID (str) – The job ID from the scheduler, if known.

• process_ID (int) – The process ID of the subprocess, if known.

• version_info (tuple[str, ...]) – Version info about the target system.

• os_name (str) – The name of the OS.

• shell_name (str) – The name of the shell.

• scheduler_name (str) – The scheduler used.

• running (bool) – Whether the jobscript is currently running.

Methods

cancel	Cancel this jobscript.
compose_jobscript	Prepare the jobscript file string.
from_json_like	Make an instance of this class from JSON (or YAML) data.
get_active_states	If this jobscript is active on this machine, return the state information from the scheduler.
get_commands_file_name	Get the name of a file containing commands for a particular jobscript action.
get_task_loop_idx_array	Get an array of task loop indices.
make_artifact_dirs	Create the directories that will hold artifacts associated with this jobscript.
submit	Submit the jobscript to the scheduler.
to_dict	Serialize this object as a dictionary.
to_json_like	Serialize this object as an object structure that can be trivially converted to JSON.
write_EAR_ID_file	Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.
write_element_run_dir_file	Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.
write_jobscript	Write the jobscript to its file.

Attributes

EAR_ID	The array of EAR IDs.
EAR_ID_file_name	The name of a file containing EAR IDs.
EAR_ID_file_path	The path to the file containing EAR IDs for this jobscript.
all_EAR_IDs	The IDs of all EARs in this jobscript.
all_EARs	Description of EAR information for this jobscript.
app
dependencies	The dependency descriptor.
direct_stderr_file_name	File for direct execution stderr.
direct_stderr_path	The path to the file containing the stderr from directly executed commands for this jobscript.
direct_stdout_file_name	File for direct execution stdout.
direct_stdout_path	The path to the file containing the stdout from directly executed commands for this jobscript.
direct_win_pid_file_name	File for holding the direct execution PID.
direct_win_pid_file_path	The path to the file containing PIDs for directly executed commands for this jobscript.
element_run_dir_file_name	The name of a file containing run directory names.
element_run_dir_file_path	The path to the file containing run directory names for this jobscript.
end_time	The last known end time of any EAR in this jobscript.
index	The index of this jobscript within its parent Submission.
is_array	Whether to generate an array job.
is_submitted	Whether this jobscript has been submitted.
jobscript_name	The name of the jobscript file.
jobscript_path	The path to the file containing the jobscript file.
num_actions	The number of actions in this jobscript.
num_elements	The number of elements in this jobscript.
os_name	The name of the OS to use.
process_ID	The process ID from direct execution, if known.
resources	The common resources that this jobscript requires.
scheduler	The scheduler that submissions go to from this jobscript.
scheduler_job_ID	The job ID from the scheduler, if known.
scheduler_js_ref	The reference to the submitted job for the jobscript.
scheduler_name	The name of the scheduler to use.
scheduler_ref	The generalised scheduler reference descriptor.
shell	The shell for composing submission scripts.
shell_name	The name of the shell to use.
start_time	The first known start time of any EAR in this jobscript.
submission	The parent submission.
submit_cmdline	The command line used to do the commit, if known.
submit_hostname	Where the jobscript was submitted, if known.
submit_machine	Description of what the jobscript was submitted to, if known.
submit_time	When the jobscript was submitted, if known.
task_actions	The IDs of actions of each task in this jobscript.
task_elements	The IDs of elements of each task in this jobscript.
task_insert_IDs	The insertion IDs of tasks in this jobscript.
task_loop_idx	The description of where various task loops are.
version_info	Version information about the execution environment (OS, etc).
workflow	The workflow this is all on behalf of.
workflow_app_alias	Alias for the workflow app in job scripts.

property EAR_ID

The array of EAR IDs.

property EAR_ID_file_name

The name of a file containing EAR IDs.

property EAR_ID_file_path

The path to the file containing EAR IDs for this jobscript.

property all_EAR_IDs: List[int]

The IDs of all EARs in this jobscript.

property all_EARs: List

Description of EAR information for this jobscript.

app = App(name='MatFlow', version='0.3.0a132')

cancel()

Cancel this jobscript.

compose_jobscript(deps=None, os_name=None, shell_name=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)

Prepare the jobscript file string.

Parameters:

• deps (Dict | None) –

• os_name (str) –

• shell_name (str) –

• os_args (Dict | None) –

• shell_args (Dict | None) –

• scheduler_name (str | None) –

• scheduler_args (Dict | None) –

Return type:

str

property dependencies

The dependency descriptor.

property direct_stderr_file_name

File for direct execution stderr.

property direct_stderr_path

The path to the file containing the stderr from directly executed commands for this jobscript.

property direct_stdout_file_name

File for direct execution stdout.

property direct_stdout_path

The path to the file containing the stdout from directly executed commands for this jobscript.

property direct_win_pid_file_name

File for holding the direct execution PID.

property direct_win_pid_file_path

The path to the file containing PIDs for directly executed commands for this jobscript. Windows only.

property element_run_dir_file_name

The name of a file containing run directory names.

property element_run_dir_file_path

The path to the file containing run directory names for this jobscript.

property end_time

The last known end time of any EAR in this jobscript.

classmethod from_json_like(json_like, shared_data=None)

Make an instance of this class from JSON (or YAML) data.

Parameters:

• json_like – The data to deserialise.

• shared_data – Shared context data.

Return type:

The deserialised object.

get_active_states(as_json=False)

If this jobscript is active on this machine, return the state information from the scheduler.

Parameters:

as_json (bool) –

Return type:

Dict[int, JobscriptElementState]

get_commands_file_name(js_action_idx, shell=None)

Get the name of a file containing commands for a particular jobscript action.

get_task_loop_idx_array()

Get an array of task loop indices.

property index

The index of this jobscript within its parent Submission.

property is_array

Whether to generate an array job.

property is_submitted

Whether this jobscript has been submitted.

property jobscript_name

The name of the jobscript file.

property jobscript_path

The path to the file containing the jobscript file.

make_artifact_dirs()

Create the directories that will hold artifacts associated with this jobscript.

property num_actions

The number of actions in this jobscript.

property num_elements

The number of elements in this jobscript.

property os_name: str | None

The name of the OS to use.

property process_ID

The process ID from direct execution, if known.

property resources

The common resources that this jobscript requires.

property scheduler

The scheduler that submissions go to from this jobscript.

property scheduler_job_ID

The job ID from the scheduler, if known.

property scheduler_js_ref

The reference to the submitted job for the jobscript.

property scheduler_name: str | None

The name of the scheduler to use.

property scheduler_ref

The generalised scheduler reference descriptor.

property shell

The shell for composing submission scripts.

property shell_name: str | None

The name of the shell to use.

property start_time

The first known start time of any EAR in this jobscript.

property submission

The parent submission.

submit(scheduler_refs, print_stdout=False)

Submit the jobscript to the scheduler.

Parameters:

• scheduler_refs (Dict[int, (str, bool)]) –

• print_stdout (Optional[bool]) –

Return type:

str

property submit_cmdline

The command line used to do the commit, if known.

property submit_hostname

Where the jobscript was submitted, if known.

property submit_machine

Description of what the jobscript was submitted to, if known.

property submit_time

When the jobscript was submitted, if known.

property task_actions

The IDs of actions of each task in this jobscript.

property task_elements

The IDs of elements of each task in this jobscript.

property task_insert_IDs

The insertion IDs of tasks in this jobscript.

property task_loop_idx

The description of where various task loops are.

to_dict()

Serialize this object as a dictionary.

to_json_like(dct=None, shared_data=None, exclude=None, path=None)

Serialize this object as an object structure that can be trivially converted to JSON. Note that YAML can also be produced from the result of this method; it just requires a different final serialization step.

property version_info

Version information about the execution environment (OS, etc).

property workflow

The workflow this is all on behalf of.

property workflow_app_alias

Alias for the workflow app in job scripts.

write_EAR_ID_file()

Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.

write_element_run_dir_file(run_dirs)

Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.

We assume a given task element’s actions all run in the same directory, but in general a jobscript “element” may cross task boundaries, so we need to provide the directory for each jobscript-element/jobscript-action combination.

Parameters:

run_dirs (List[List[Path]]) –

write_jobscript(os_name=None, shell_name=None, deps=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)

Write the jobscript to its file.

Parameters:

• os_name (str) –

• shell_name (str) –

• deps (Dict | None) –

• os_args (Dict | None) –

• shell_args (Dict | None) –

• scheduler_name (str | None) –

• scheduler_args (Dict | None) –