matflow.Workflow#
- class matflow.Workflow(workflow_ref, store_fmt=None, fs_kwargs=None, **kwargs)#
Bases:
WorkflowA concrete workflow.
- Parameters:
workflow_ref (str | Path | int) – A path to a persistent workflow, a registered workflow name, or a local submission ID reported by the app show command. Path objects and explicit path strings are always interpreted as paths. The ID
-1selects the most recently submitted registered workflow.store_fmt (str | None) – The format of persistent store to use. Used to select the store manager class.
fs_kwargs (dict[str, Any] | None) – Additional arguments to pass when resolving a virtual workflow reference.
kwargs – For compatibility during pre-stable development phase.
Methods
Abort the currently running action-run of the specified task/element.
Add a loop to a subset of workflow tasks.
Add a new submission.
Add a task to this workflow.
Add a new task after the specified task.
Add a new task before the specified task.
A context manager that batches up structural changes to the workflow and commits them to disk all together when the context manager exits.
Cancel any running jobscripts.
Check if all the parameters exist.
Merge each output's shard archives into one indexed
consolidated.zip.Copy the workflow to a new path and return the copied workflow path.
Delete the persistent data.
Get the elements of the workflow's tasks.
Ensure a commands file exists for the specified run.
Execute a combined script (multiple runs) via a subprocess.
Execute commands of a run via a subprocess, using the parameter metadata cache.
Export the selected array output to a single NumPy
.npyfile.Generate from a JSON file.
Generate from a JSON string.
Generate from a YAML file.
Generate from a YAML string.
Generate from either a YAML or JSON file, depending on the file extension.
Generate from a WorkflowTemplate object.
Generate from the data associated with a WorkflowTemplate object.
Get EAR IDs belonging to multiple tasks.
Check if an EAR is to be skipped.
Get element action run objects from a list of IDs.
Get EARs belonging to multiple tasks.
Get all runs in the workflow.
Get all iterations in the workflow.
Get all elements in the workflow.
Retrieve all workflow parameter data.
Retrieve all persistent parameters sources.
Retrieve all persistent parameters.
Get the run IDs of all submissions.
Read a saved array output for one run ID or an iterable of run IDs.
Get the element IDs of EARs.
Get the element iteration IDs of EARs.
Return element iteration objects from a list of IDs.
Get element iterations belonging to multiple tasks.
Return element objects from a list of IDs.
Get the iteration task pathway.
Get a single parameter.
Get the data relating to a parameter.
Get whether some parameters are set.
Get the source of a particular parameter.
Get parameter sources known to the workflow.
Get parameters known to the workflow.
Return jobscript process IDs from all submissions of this workflow.
Resolve working directories using each run's
scratchresource.Retrieve elements that are running according to the scheduler.
Retrieve runs that are running according to the scheduler.
Return jobscript scheduler job IDs from all submissions of this workflow.
Get the persistent element action runs.
Get the persistent element iterations.
Get the persistent elements.
Get the persistent tasks.
Get the task IDs of elements.
Get the elements of a task.
Return the unique names of all workflow tasks.
Retrieve the contents of a text file stored within the workflow.
Test if a particular parameter is set.
Return sorted names of outputs with array archives.
Print a table listing jobscripts and associated information from the specified submission.
Print a table listing the jobscripts associated with the specified (or all) tasks for the specified submission.
Process the shell stdout/stderr stream according to the associated Command object.
Rechunk metadata/runs and parameters/base arrays, making them more efficient.
Reorganise the stored data chunks for parameters to be more efficient.
Reorganise the stored data chunks for EARs to be more efficient.
Reload the workflow from disk.
Resolve this workflow to a set of jobscripts to run for a new submission.
Save a parameter where an EAR can find it.
Set the end time and exit code on an EAR.
Record that an EAR is to be skipped due to an upstream failure or loop termination condition being met.
Set the start time on an EAR.
Set
EARs_initialisedto True for the specified iteration.Set end times and exit codes on multiple runs.
Set the start time on multiple runs.
Set the value of a parameter.
Print a description of the status of every element action run in the workflow.
Print individual action runs, optionally filtered by engine state.
Submit the workflow for execution.
Rename an existing same-path workflow (directory) so we can restore it if workflow creation fails.
Convert the workflow to an unzipped form.
Wait for specified/all active jobscripts, optionally following their output.
Convert the workflow to a zipped form.
Attributes
Path to artifacts of the workflow (temporary files, etc).
The creation descriptor for the workflow.
Default working-directory root, used when
scratchis unspecified.The ID of this workflow.
Path to input files for the workflow.
The loops in this workflow.
The name of the workflow.
The total number of element action runs.
The total number of added tasks.
The total number of element iterations.
The total number of elements.
The total number of loops.
The total number of job submissions.
The total number of tasks.
The format of the workflow's persistent store.
The job submissions done by this workflow.
Path to submission data for ths workflow.
Path to artifacts of tasks.
The tasks in this workflow.
The template that this workflow was made from.
The template components used for this workflow.
The timestamp format.
The timestamp format for names.
An fsspec URL for this workflow.
- abort_run(submission_idx=-1, task_idx=None, task_insert_ID=None, element_idx=None)#
Abort the currently running action-run of the specified task/element.
- add_loop(loop)#
Add a loop to a subset of workflow tasks.
- Parameters:
loop (Loop) –
- Return type:
None
- add_submission(tasks=None, JS_parallelism=None, force_array=False, min_jobscripts=True, status=True, timeit=False)#
Add a new submission.
- Parameters:
force_array (bool) – Used to force the use of job arrays, even if the scheduler does not support it. This is provided for testing purposes only.
min_jobscripts (bool) – If True (the default), minimise the total number of jobscripts by performing as many merges as possible. This may merge otherwise independent jobscripts, such that they are run sequentially rather than in parallel.
timeit (bool) – Time run execution function pathways as the code executes and write out a summary to the app-std file. Only functions decorated by TimeIt.decorator are included.
JS_parallelism (bool | Literal['direct', 'scheduled'] | None) –
status (bool) –
- Return type:
Submission | None
- add_task(task, new_index=None)#
Add a task to this workflow.
- add_task_after(new_task, task_ref=None)#
Add a new task after the specified task.
- add_task_before(new_task, task_ref=None)#
Add a new task before the specified task.
- property artifacts_path: Path#
Path to artifacts of the workflow (temporary files, etc).
- batch_update(is_workflow_creation=False)#
A context manager that batches up structural changes to the workflow and commits them to disk all together when the context manager exits.
- Parameters:
is_workflow_creation (bool) –
- Return type:
Iterator[None]
- cached_merged_parameters()#
- cancel(status=True, quiet=False)#
Cancel any running jobscripts.
- check_parameters_exist(id_lst)#
Check if all the parameters exist.
- consolidate_array_outputs(names=None, *, status=True, temp_dir=None)#
Merge each output’s shard archives into one indexed
consolidated.zip.For every selected output, all shard archives (
.submissions/<submission>/<shard>/<file>.npz.zst, plus any legacy<shard>/<file>.npz.zst) and any existing consolidated archive are merged, sorted by run ID, written atomically toparameters/array_outputs/<name>/consolidated.zip, verified one chunk at a time, and only then are the shard files removed. Runs saved later go to new shard archives, so consolidation may be repeated to fold them in. Readers (get_array_output(), parameter decoding) handle both forms transparently. Mixed shapes are stored in independently compressed chunks without global padding, targeting 16 MiB per chunk (larger entries get their own chunk). Existing NPZ/Zstandard archives remain readable and are migrated on consolidation, including when no new shards exist. A metadata pass is followed by a payload pass, opening one source at a time and buffering at most 32 MiB of shape chunks, plus source entries and metadata. Legacy NPZs spill to temporary disk above 8 MiB; nonstandard compressed/Fortran NPYs can still load one source array. Older hpcflow versions cannot read the indexed format.- Parameters:
names (str | Iterable[str] | None) – Output name(s) to consolidate (
outputs.prefix optional).Noneconsolidates every output with array archives. Masked-array mask archives (<name>.__mask__) are consolidated with their data.status (bool) – Show a live status while consolidating.
temp_dir (str | Path | None) – Existing directory for source NPZ buffers that spill to disk.
Noneuses a unique subdirectory ofparameters/array_outputs/.tmp/. Each output’s temporary subdirectory is removed on success or failure; the empty.tmpparent is retained for concurrent operations. The replacement archive remains beside its destination so the final rename stays atomic.
- Returns:
Mapping from output name to its consolidated archive path. Outputs that had
nothing to consolidate (already consolidated) map to the existing path.
- Raises:
TypeError – If the store is read-only (e.g. a zipped workflow), or shards of one output have different dtypes.
ValueError – If a named output has no archives, shards differ in dimensionality, or a a run/member pair appears more than once.
- Return type:
- copy(path='.')#
Copy the workflow to a new path and return the copied workflow path.
- Parameters:
path (str | Path) –
- Return type:
Path
- property creation_info: CreationInfo#
The creation descriptor for the workflow.
- delete()#
Delete the persistent data.
- Return type:
None
- ensure_commands_file(submission_idx, js_idx, run)#
Ensure a commands file exists for the specified run.
- Parameters:
submission_idx (int) –
js_idx (int) –
run (ElementActionRun) –
- Return type:
Path | bool
- execute_combined_runs(submission_idx, jobscript_idx)#
Execute a combined script (multiple runs) via a subprocess.
- execute_run(submission_idx, block_act_key, run_ID)#
Execute commands of a run via a subprocess, using the parameter metadata cache.
- property execution_path: Path#
Default working-directory root, used when
scratchis unspecified.
- export_array_output(name, run_IDs=None, path=None, *, sort=True, overwrite=False)#
Export the selected array output to a single NumPy
.npyfile.Selection, sorting, and scalar-run shape follow
get_array_output(). List-backed results are stacked if all selected shapes match; otherwise raise ValueError before writing. Write to exactlypath(no automatic suffix) and return that path.pathis required; omitrun_IDsto export every saved run. Existing files raise FileExistsError unlessoverwrite=True.
- classmethod from_JSON_file(JSON_path, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, status=None)#
Generate from a JSON file.
- Parameters:
JSON_path (PathLike) – The path to a workflow template in the JSON file format.
path (PathLike) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the file given by JSON_path. Substitutions will be attempted if the JSON file looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!
updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.
resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.
config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.
status (Status | None) –
- Return type:
- classmethod from_JSON_string(JSON_str, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, status=None)#
Generate from a JSON string.
- Parameters:
JSON_str (str) – The JSON string containing a workflow template parametrisation.
path (PathLike) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the string JSON_str. Substitutions will be attempted if the JSON string looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!
updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.
resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.
config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.
status (Status | None) –
- Return type:
- classmethod from_YAML_file(YAML_path, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, status=None)#
Generate from a YAML file.
- Parameters:
YAML_path (PathLike) – The path to a workflow template in the YAML file format.
path (PathLike) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the file given by YAML_path. Substitutions will be attempted if the YAML file looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!
updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.
resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.
config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.
status (Status | None) –
- Return type:
- classmethod from_YAML_string(YAML_str, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, status=None)#
Generate from a YAML string.
- Parameters:
YAML_str (str) – The YAML string containing a workflow template parametrisation.
path (PathLike) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the string YAML_str. Substitutions will be attempted if the YAML string looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!
updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.
resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.
config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.
status (Status | None) –
- Return type:
- classmethod from_file(template_path, template_format=None, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, status=None)#
Generate from either a YAML or JSON file, depending on the file extension.
- Parameters:
template_path (PathLike) – The path to a template file in YAML or JSON format, and with a “.yml”, “.yaml”, or “.json” extension.
template_format (Literal['json', 'yaml'] | None) – If specified, one of “json” or “yaml”. This forces parsing from a particular format regardless of the file extension.
path (str | None) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the file given by template_path. Substitutions will be attempted if the file looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!
updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.
resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.
config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.
status (Status | None) –
- Return type:
- classmethod from_template(template, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, status=None)#
Generate from a WorkflowTemplate object.
- Parameters:
template (WorkflowTemplate) – The WorkflowTemplate object to make persistent.
path (PathLike) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with config the item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
status (Status | None) –
- Return type:
- classmethod from_template_data(template_name, imports=None, tasks=None, loops=None, resources=None, environments=None, config=None, path=None, workflow_name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, inputs=None)#
Generate from the data associated with a WorkflowTemplate object.
- Parameters:
template_name (str) – The name to use for the new workflow template, from which the new workflow will be generated.
imports (list[Import] | None) – A list of imports objects to be used in the workflow.
tasks (list[Task] | None) – List of Task objects to add to the new workflow.
loops (list[Loop] | None) – List of Loop objects to add to the new workflow.
inputs (list[InputValue] | dict[str, Any] | None) – Shared workflow-template input values. Task sources take precedence.
resources (Resources) – Mapping of action scopes to resource requirements, to be applied to all element sets in the workflow. resources specified in an element set take precedence of those defined here for the whole workflow.
environments (Mapping[str, Mapping[str, Any]] | None) – Environment specifiers, keyed by environment name.
config (dict | None) – Configuration items that should be set whenever the resulting workflow is loaded. This includes config items that apply during workflow execution.
path (PathLike | None) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
workflow_name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool | None) – If True, suffix the workflow name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool | None) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.
- Return type:
- get_EAR_IDs_of_tasks(id_lst)#
Get EAR IDs belonging to multiple tasks.
- get_EAR_skipped(EAR_ID)#
Check if an EAR is to be skipped.
- get_EARs_from_IDs(ids, as_dict=False)#
Get element action run objects from a list of IDs.
- Parameters:
- Return type:
list[ElementActionRun] | dict[int, ElementActionRun] | ElementActionRun
- get_EARs_of_tasks(id_lst)#
Get EARs belonging to multiple tasks.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Iterator[ElementActionRun]
- get_all_EARs()#
Get all runs in the workflow.
- Return type:
- get_all_element_iterations()#
Get all iterations in the workflow.
- Return type:
- get_all_parameter_data(**kwargs)#
Retrieve all workflow parameter data.
- get_all_parameter_sources(**kwargs)#
Retrieve all persistent parameters sources.
- Return type:
list[ParamSource]
- get_all_parameters(**kwargs)#
Retrieve all persistent parameters.
- get_array_output(name, run_IDs=None, *, sort=True)#
Read a saved array output for one run ID or an iterable of run IDs.
Return an
ArrayOutputwith data and all per-run metadata selected including repeated IDs. By default, sort lexicographically by task insert ID, element index, iteration index, action index, run index, then member index. Equal keys retain their requested order. Withsort=False, preserve the requested order. Data, metadata, and shapes are reordered together. For a scalar ID,datais that run’s array for ordinary outputs. Member outputs retain their leading member axis even for a scalar run ID, with original indices inmember_idx. For multiple entries, data is stacked when no shape metadata is present and all shapes match, otherwise it is a list. Sparse sub-parameters contain only members that supplied that array.New Zarr array outputs are stored as indexed per-run records in
.membersarchives, optionally merged into oneconsolidated.zipper output byconsolidate_array_outputs(). Reads can select one array path without loading unrelated large arrays. Existing per-path NPZ/Zstandard archives and version-1 member archives remain readable and are migrated when consolidated. Withrun_IDs=None, include every saved run for this output, across tasks and iterations. Withsort=False, archives are visited in path order.
- get_element_IDs_from_EAR_IDs(id_lst)#
Get the element IDs of EARs.
- get_element_iteration_IDs_from_EAR_IDs(id_lst)#
Get the element iteration IDs of EARs.
- get_element_iterations_from_IDs(id_lst)#
Return element iteration objects from a list of IDs.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
- get_element_iterations_of_tasks(id_lst)#
Get element iterations belonging to multiple tasks.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Iterator[ElementIteration]
- get_elements_from_IDs(id_lst)#
Return element objects from a list of IDs.
- get_iteration_task_pathway(ret_iter_IDs=False, ret_data_idx=False)#
Get the iteration task pathway.
- get_parameter(index, **kwargs)#
Get a single parameter.
Parameter#
- index:
The index of the parameter to retrieve.
- keyword dataset_copy:
For Zarr stores only. If True, copy arrays as NumPy arrays.
- kwtype dataset_copy:
bool
- Parameters:
index (int) –
- Return type:
StoreParameter
- get_parameter_data(index, **kwargs)#
Get the data relating to a parameter.
- Parameters:
index (int) –
- Return type:
Any
- get_parameter_set_statuses(id_lst)#
Get whether some parameters are set.
An output-member reference is set exactly when its producing whole output is set.
- get_parameter_source(index)#
Get the source of a particular parameter.
- Parameters:
index (int) –
- Return type:
ParamSource
- get_parameter_sources(id_lst)#
Get parameter sources known to the workflow.
An output-member reference has the same source as its producing whole output; its member index is metadata for value resolution, not a separate producer.
- get_parameters(id_lst, **kwargs)#
Get parameters known to the workflow.
Output-member references are resolved here to the indexed value from their producing whole output. The member itself remains a deferred reference in persistent storage, so callers should use this API rather than reading the backing store directly.
Parameter#
- id_lst:
The indices of the parameters to retrieve.
- keyword dataset_copy:
For Zarr stores only. If True, copy arrays as NumPy arrays.
- kwtype dataset_copy:
bool
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Sequence[StoreParameter]
- get_process_IDs()#
Return jobscript process IDs from all submissions of this workflow.
- get_run_directories(run_ids=None, dir_indices_arr=None)#
Resolve working directories using each run’s
scratchresource.No directories are created. If
dir_indices_arris supplied, its rows must correspond torun_ids(or the first runs, ifrun_idsis omitted).
- get_running_elements(submission_idx=-1, task_idx=None, task_insert_ID=None)#
Retrieve elements that are running according to the scheduler.
- get_running_runs(submission_idx=-1, task_idx=None, task_insert_ID=None, element_idx=None)#
Retrieve runs that are running according to the scheduler.
- get_scheduler_job_IDs()#
Return jobscript scheduler job IDs from all submissions of this workflow.
- get_store_EARs(id_lst)#
Get the persistent element action runs.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Sequence[StoreEAR]
- get_store_element_iterations(id_lst)#
Get the persistent element iterations.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Sequence[StoreElementIter]
- get_store_elements(id_lst)#
Get the persistent elements.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Sequence[StoreElement]
- get_store_tasks(id_lst)#
Get the persistent tasks.
- Parameters:
id_lst (Iterable[int]) –
- Return type:
Sequence[StoreTask]
- get_task_IDs_from_element_IDs(id_lst)#
Get the task IDs of elements.
- get_task_elements(task, idx_lst=None)#
Get the elements of a task.
- Parameters:
task (WorkflowTask) –
- Return type:
- get_task_unique_names(map_to_insert_ID=False)#
Return the unique names of all workflow tasks.
- get_text_file(path)#
Retrieve the contents of a text file stored within the workflow.
- property input_files_path: Path#
Path to input files for the workflow.
- is_parameter_set(index)#
Test if a particular parameter is set.
- list_array_outputs()#
Return sorted names of outputs with array archives.
Both shard archives (
.submissions/<submission>/<shard>/<file>.npz.zst, or the legacy<shard>/<file>.npz.zst) and consolidated archives (seeconsolidate_array_outputs()) are considered. Internal__mask__archives (masked-array masks) are excluded.
- list_jobscripts(sub_idx=0, max_js=None, jobscripts=None, width=None)#
Print a table listing jobscripts and associated information from the specified submission.
- Parameters:
sub_idx (int) – The submission index whose jobscripts are to be displayed.
max_js (int | None) – Maximum jobscript index to display. This cannot be specified with jobscripts.
jobscripts (list[int] | None) – A list of jobscripts to display. This cannot be specified with max_js.
width (int | None) – Width in characters of the printed table.
- Return type:
None
- list_task_jobscripts(sub_idx=0, task_names=None, max_js=None, width=None)#
Print a table listing the jobscripts associated with the specified (or all) tasks for the specified submission.
- Parameters:
sub_idx (int) – The submission index whose jobscripts are to be displayed.
task_names (list[str] | None) – List of sub-strings to match to task names. Only matching task names will be included.
max_js (int | None) – Maximum jobscript index to display.
width (int | None) – Width in characters of the printed table.
- property loops: WorkflowLoopList#
The loops in this workflow.
- property name: str#
The name of the workflow.
The workflow name may be different from the template name, as it includes the creation date-timestamp if generated.
- process_shell_parameter_output(name, value, EAR_ID, cmd_idx, stderr=False)#
Process the shell stdout/stderr stream according to the associated Command object.
- rechunk(chunk_size=None, backup=True, status=True)#
Rechunk metadata/runs and parameters/base arrays, making them more efficient.
- rechunk_parameter_base(chunk_size=None, backup=True, status=True)#
Reorganise the stored data chunks for parameters to be more efficient.
- rechunk_runs(chunk_size=None, backup=True, status=True)#
Reorganise the stored data chunks for EARs to be more efficient.
- reload()#
Reload the workflow from disk.
- Return type:
Self
- resolve_jobscripts(cache, tasks=None, force_array=False, min_jobscripts=True)#
Resolve this workflow to a set of jobscripts to run for a new submission.
- Parameters:
force_array (bool) – Used to force the use of job arrays, even if the scheduler does not support it. This is provided for testing purposes only.
min_jobscripts (bool) – If True (the default), minimise the total number of jobscripts by performing as many merges as possible. This may merge otherwise independent jobscripts, such that they are run sequentially rather than in parallel.
cache (ObjectCache) –
tasks (Sequence[int] | None) –
- Return type:
- save_parameter(name, value, EAR_ID)#
Save a parameter where an EAR can find it.
With the Zarr store, output ndarrays use Zstandard-compressed NPZ archives instead of legacy array datasets unless excluded by task/schema
disable_array_outputs. This is a temporary storage workaround. The JSON store retains its existing encoding and does not write NPZ archives.
- set_EAR_end(block_act_key, run, exit_code)#
Set the end time and exit code on an EAR.
If the exit code is non-zero, also set all downstream dependent EARs to be skipped. Also save any generated input/output files.
- Parameters:
block_act_key (BlockActionKey) –
run (ElementActionRun) –
exit_code (int) –
- Return type:
- set_EAR_skip(skip_reasons)#
Record that an EAR is to be skipped due to an upstream failure or loop termination condition being met.
- Parameters:
skip_reasons (dict[int, hpcflow.sdk.core.skip_reason.SkipReason]) –
- Return type:
None
- set_EAR_start(run_id, run_dir, port_number)#
Set the start time on an EAR.
- set_EARs_initialised(iter_ID)#
Set
EARs_initialisedto True for the specified iteration.- Parameters:
iter_ID (int) –
- Return type:
None
- set_multi_run_ends(runs)#
Set end times and exit codes on multiple runs.
If the exit code is non-zero, also set all downstream dependent runs to be skipped. Also save any generated input/output files.
- set_multi_run_starts(run_ids, run_dirs, port_number)#
Set the start time on multiple runs.
- set_parameter_value(param_id, value, commit=False)#
Set the value of a parameter.
- set_parameter_values(values, commit=False)#
- show_all_EAR_statuses()#
Print a description of the status of every element action run in the workflow.
- Return type:
None
- show_runs(state=None, *, task_idx=None, task_id=None, task_name=None, element_idx=None, iteration_idx=None, action_idx=None, run_id=None, run_idx=None, columns=None)#
Print individual action runs, optionally filtered by engine state.
- Parameters:
state (EARStatus or str, optional) – An engine state or its case-insensitive name, such as
error. Omit to show all runs. Invalid names raise ValueError.task_idx (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.task_id (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.element_idx (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.iteration_idx (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.action_idx (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.run_id (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.run_idx (IndexFilter) – Optional non-negative index/ID selections: an integer, integer sequence, or comma-separated indices and inclusive ranges such as
0,2-5. Combine coordinates with AND. Task indices are positions in workflow order; task IDs are stable insertion IDs. Run IDs are global, while run indices identify per-action attempts. Other indices are zero-based.task_name (str | Sequence[str] | None) – Exact task unique name(s), comma-separated or a sequence.
columns (str | Sequence[str] | None) – Ordered column identifiers, comma-separated or a sequence.
allselects every column. Optional fields include host, start, end, duration, and array_idx (zero-based scheduler array item; absent for serial jobs). end_size and end_size_bytes sum regular-file sizes in snapshot_end, using readable units or exact bytes without scanning the run directory. Datetimes use show_timezone. Running durations use the current time.
- Return type:
None
Notes
Include all attempts across iterations and submissions, so failed runs remain inspectable after successful retries. Display run IDs, task and element coordinates, action short names, attempt/submission/jobscript indices, states, and exit codes. Read workflow data without querying schedulers or updating monitoring files.
- property submissions: list[hpcflow.sdk.submission.submission.Submission]#
The job submissions done by this workflow.
- property submissions_path: Path#
Path to submission data for ths workflow.
- submit(*, ignore_errors=False, JS_parallelism=None, min_jobscripts=True, modify_js=False, print_stdout=False, wait=False, add_to_known=True, return_idx=False, tasks=None, cancel=False, status=True, quiet=False, timeit=False, follow=False, delete=False)#
Submit the workflow for execution.
- Parameters:
ignore_errors (bool) – If True, ignore jobscript submission errors. If False (the default) jobscript submission will halt when a jobscript fails to submit.
JS_parallelism (bool | Literal['direct', 'scheduled'] | None) – If True, allow multiple jobscripts to execute simultaneously. If ‘scheduled’/’direct’, only allow simultaneous execution of scheduled/direct jobscripts. Raises if set to True, ‘scheduled’, or ‘direct’, but the store type does not support the jobscript_parallelism feature. If not set, jobscript parallelism will be used if the store type supports it, for scheduled jobscripts only.
min_jobscripts (bool) – If True (the default), minimise the total number of jobscripts by performing as many merges as possible. This may merge otherwise independent jobscripts, such that they are run sequentially rather than in parallel.
modify_js (bool) – If True, pause before submitting each jobscript, print the new jobscript’s path and await confirmation before continuing submission. This allows ad hoc modifications to the jobscript to be made.
print_stdout (bool) – If True, print any jobscript submission standard output, otherwise hide it.
wait (bool) – If True, this command will block until the workflow execution is complete.
follow (bool) – If True, follow jobscript stdout and stderr until execution completes. Implies wait. Ctrl+C stops following without cancelling execution. Only direct schedulers are supported; other schedulers are rejected before any jobscripts are submitted.
delete (bool) – If True, permanently delete the workflow, without confirmation, after execution completes, whether or not it succeeded. Implies wait. If waiting is interrupted (e.g. by Ctrl+C), the workflow is not deleted. If cancel is also True, wait for the cancelled jobscripts to stop before deleting.
add_to_known (bool) – If True, add the submitted submissions to the known-submissions file, which is used by the show command to monitor current and recent submissions.
return_idx (bool) – If True, return a dict representing the jobscript indices submitted for each submission.
tasks (list[int] | None) – List of task indices to include in the new submission if no submissions already exist. By default all tasks are included if a new submission is created.
cancel (bool) – Immediately cancel the submission. Useful for testing and benchmarking.
status (bool) – If True, display a live status to track submission progress.
quiet (bool) – If True, do not print messages about the workflow submission.
timeit (bool) – Time run execution function pathways as the code executes and write out a summary to the app-std file. Only functions decorated by TimeIt.decorator are included.
- Return type:
- property task_artifacts_path: Path#
Path to artifacts of tasks.
- property tasks: WorkflowTaskList#
The tasks in this workflow.
- property template: WorkflowTemplate#
The template that this workflow was made from.
- property template_components: TemplateComponents#
The template components used for this workflow.
- classmethod temporary_rename(path, fs)#
Rename an existing same-path workflow (directory) so we can restore it if workflow creation fails.
Renaming will occur until the successfully completed. This means multiple new paths may be created, where only the final path should be considered the successfully renamed workflow. Other paths will be deleted.
- unzip(path='.', *, log=None)#
Convert the workflow to an unzipped form.
- wait(sub_js=None, quiet=False, *, follow=False)#
Wait for specified/all active jobscripts, optionally following their output.
With follow=True, print existing output from the beginning, then stream new output until all selected jobscripts finish. Separate error streams are written to stderr; combined streams are written once to stdout. quiet hides status messages and stream headers, but not job output. Following is supported only for direct schedulers. Selecting a non-direct jobscript raises NotImplementedError without polling its scheduler.
- zip(path='.', *, log=None, overwrite=False, include_execute=False, include_rechunk_backups=False, consolidate=True, status=True, temp_dir=None)#
Convert the workflow to a zipped form.
- Parameters:
path (str) – Path at which to create the new zipped workflow. If this is an existing directory, the zip file will be created within this directory. Otherwise, this path is assumed to be the full file path to the new zip file.
consolidate (bool) – If True (default), first consolidate array-output shard archives into a single file per output (see
consolidate_array_outputs()); zipping many small shard files is much slower than zipping one file per output.temp_dir (str | Path | None) – Existing directory for decompressed source buffers during consolidation. Defaults to a unique subdirectory of
parameters/array_outputs/.tmp/. Temporary subdirectories are cleaned up afterward. Replacement archives are created beside their destination to preserve atomic replacement.log (str | None) –
overwrite (bool) –
include_execute (bool) –
include_rechunk_backups (bool) –
status (bool) –
- Return type: