Demo workflows

A good way to get started with MatFlow is to run one of the built-in demo workflows. This will also test your installation, configuration, and some of your environments.

Submit a workflow

MatFlow comes with some demo workflows, which can be listed using

matflow demo-workflow --list

We can run the following command to copy the in-built workflow file to the current directory (note the final dot at the end),

matflow demo-workflow copy modify_volume_element_grid_size .

which we can then use to submit the workflow.

matflow go modify_volume_element_grid_size.yaml

This small workflow should complete in less than 30s. Note that there is also a convenience shortcut for the demo workflows which combines the copy-then-submit pattern we saw above:

matflow demo-workflow go modify_volume_element_grid_size

However, in general workflows would first be created in a yaml file which is then submitted using matflow go WORKFLOW_FILE.

Check the status of a workflow

After submitting a workflow, you can check whether it has run successfully using

matflow show -f

For clarification of the output, a legend can be shown using

matflow show --legend

Cancel a workflow

Sometimes you might want to cancel a workflow that is running. Use

matflow cancel WORKFLOW_REF

where WORKFLOW_REF is either the path to the workflow directory, or the ID of the workflow displayed by matflow show.

Workflow output

In your working directory, a new directory will be created with the name of the workflow and the date and time of submission. This directory contains the output of the workflow, including your results files, the log files and any other files generated by the workflow.

The directory structure is as follows:

workflow_name_YYYY-MM-DD_timestamp/
├── artifacts/
│   ├── workflow.yaml
│   ├── submissions/
│   └── tasks/
├── execute/
├── metadata/
└── parameters/

The artifacts directory contains a copy of the workflow template file that you submitted, the submission files, and the task artifact files. The submissions directory contains generated jobscripts that are submitted to the scheduler, standard output and error stream files from those jobscripts, and app log files (if MatFlow is configured to generate them). If your workflow crashes, you should first check the files in the js_std directory to see what happened. The tasks directory contains files associated with task execution that we wish to keep (for example input and output files for a simulation). We can control which files are kept within the task schema.

The execute directory contains the working directories for the tasks, and so any output files generated by the software that is invoked will be here. The metadata and parameters directories contain binary files, corresponding to workflow structural metadata, and input and output parameters, respectively. You will not normally need to look at these files, but they are available there for advanced debugging and checkpointing.