Secrets

MatFlow can store secrets like usernames and passwords for third party services that require authentication.

Warning

Do not store secrets like usernames, passwords, and API keys within environment definitions. These definitions are copied into workflow metadata and so would be distributed with the workflow, if shared.

Managing secrets

Secrets can be retrieved, set, and deleted as follows:

CLI

Using the manage secrets sub-command in the MatFlow CLI, we can set a new secret like this:

matflow manage secrets set KEY VALUE

To update the value of an existing secret, use the –overwrite flag:

matflow manage secrets set KEY VALUE --overwrite

Items can be retrieved like this:

matflow manage secrets get KEY

All currently defined secrets can be listed like this (with values hidden):

matflow manage secrets list

To show all secret keys and their secret values, use the –values or -v option:

matflow manage secrets list --values

To delete a secret:

matflow manage secrets delete KEY

Python API

import matflow as mf

# set the value of the `KEY` secret:
mf.set_secret("KEY", "VALUE")

# update the existing value of the `KEY` secret:
mf.set_secret("KEY", "VALUE", overwrite=True)

# retrieve a secret's value:
print(mf.get_secret("KEY"))

# print all secret keys:
mf.print_secrets()

# print all secrets keys and their secret values:
mf.print_secrets(include_values=True)

# delete a secret
mf.delete_secret("KEY")

Note

MatFlow does not encrypt secrets as there is not a straightforward way to do so across all platforms, and for typical use cases (like running on HPC systems). Instead, MatFlow stores secrets in a file with appropriate permissions set such that the file can only be read (or written to) by the current user. The secrets file is stored within the app’s user data directory.

Using secrets

You can expose secrets as shell environment variables by specifying a list of secret keys (i.e. the names of the secrets you wish to include) in the environment definition, via the secrets argument. (As stated above, secrets values should not be stored within environment definition setup or command blocks.). For example, such an environment definition might look like this in YAML:

- name: my_env
  secrets:
    - MY_SECRET
  executables:
    - label: my_executable
      instances:
        - command: my_executable.exe
          parallel_mode: null
          num_cores:
            start: 1
            stop: 4

where MY_SECRET is the key to a secret stored by MatFlow, such that if you run matflow manage secrets get MY_SECRET, the secret’s value would be printed.