Environments#

matlab_env#

  • There are two ways of running tasks that use MTEX. Scripts can either be compiled and then the compiled application can be run, or the script can be run directly.

  • This is controlled by the compile input parameter, which is False by default. When compile is False, the run_mtex executable must be defined in the matlab_env. When compile is True, the compile_mtex and run_compiled_mtex exectuables must both be fined in the matlab_env.

  • In the examples below, all executables are defined, meaning MTEX tasks can be run with compile=True or compile=False.

  • For direct script execution (compiled=False), the MATLAB -batch switch is used, and is documented here for Windows, MacOS, and Linux.

  • Note: use of the compile_mtex executable requires that the Matlab Compiler add-on is installed, which can be performed via the Add-on explorer within the Matlab GUI.

  • TODO: This is currently tested only on Windows

Example environment definition - Windows#

- name: matlab_env
  executables:

    - label: run_mtex
      instances:
        - command: |
            & 'C:\path\to\matlab.exe' -batch "<<script_name_no_ext>> <<args>>"
          num_cores: 1
          parallel_mode: null

    - label: compile_mtex
      instances:
        - command: |
            $mtex_path = 'C:\path\to\mtex\folder'
            $mtex_include = ((Get-ChildItem -Recurse -Directory -Path $mtex_path | %{ "-I `"$_.FullName`"" }) -join ' ') + " -a `"$mtex_path\data`""
            $mtex_include | & 'C:\path\to\mcc.bat' -R -singleCompThread -m .\<<script_name>> <<args>>
          num_cores: 1
          parallel_mode: null

    - label: run_compiled_mtex
      instances:
        - command: .\<<script_name>>.exe <<args>>
          num_cores: 1
          parallel_mode: null

Example environment definition - Linux/MacOS#

- name: matlab_env
  setup: |
    # set up commands (e.g. `module load ...`)
  executables:

    - label: run_mtex
      instances:
        - command: |
            & 'C:\path\to\matlab.exe' -batch "<<script_name_no_ext>> <<args>>"
          num_cores: 1
          parallel_mode: null

    - label: compile_mtex
      instances:
        - command: compile-mtex <<script_name>> <<args>> # TODO - define this
          num_cores: 1
          parallel_mode: null

    - label: run_compiled_mtex
      instances:
        - command: |
            MATLAB_DIR=/path/to/matlab/runtime/directory
            ./run_<<script_name>>.sh $MATLAB_DIR <<args>>
          num_cores: 1
          parallel_mode: null

dream_3D_env#

Two executables are required:

  • dream_3D_runner: this is the pipeline runner which processes a pipeline.json file.

  • python_script: this is used to generate the pipeline.json file using a Python script.

Example environment definition - Linux/MacOS#

- name: dream_3D_env
  executables:
    - label: dream_3D_runner
      instances:
        - command: /path/to/DREAM3D-directory/bin/PipelineRunner
          num_cores: 1
          parallel_mode: null
    - label: python_script
      instances:
        - command: python <<script_name>> <<args>>
          num_cores: 1
          parallel_mode: null

Example environment definition - Windows#

- name: dream_3D_env
  executables:
    - label: dream_3D_runner
      instances:
        - command: "& 'C:\\path\\to\\DREAM3D-directory\\PipelineRunner.exe'"
          num_cores: 1
          parallel_mode: null
    - label: python_script
      instances:
        - command: python <<script_name>> <<args>>
          num_cores: 1
          parallel_mode: null

defdap_env#

  • The included DefDap scripts currently work only with DefDAP version 0.93.4 and up to Numpy version 1.23.5.

damask_parse#

We used our CentOS docker image (orgs/hpcflow) to produce a “relocatable” conda environment for the damask_parse MatFlow environment, using conda-pack. Using the CentOS image is required because of glibc compatibilities.

In the container:

  • Install Miniconda via the bash installation script: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html

  • Initialise conda for use in the shell: conda init

  • Reload .bashrc: source ~/.bashrc

  • Install conda pack: conda install conda-pack

  • Create a new conda environment that contains damask-parse and matflow: conda create -n matflow_damask_parse_v3a7_env python=3.10

  • Install libGL for VTK (required by the damask python package) yum install mesa-libGL

  • Activate the environment: conda activate matflow_damask_parse_v3a7_env

  • Add packages via pip: pip install matflow-new damask-parse

  • Deactivate the environment: conda deactivate

  • Pack the environment into a tarball: conda pack matflow_damask_parse_v3a7_env

  • Save the resulting compressed file outside of the container and transfer to the target machine

On the target machine:

  • Unpack the environment:

    mkdir matflow_damask_parse_v3a7_env
tar -xzf matflow_damask_parse_v3a7_env.tar.gz -C matflow_damask_parse_v3a7_env

  • Activate the environment: source matflow_damask_parse_v3a7_env/bin/activate

  • Run: conda-unpack

  • The environment can now be activated as normal using the source command above.

Resources:

Example environment definition#

name: damask_parse_env
setup: |
  conda activate matflow_damask_parse_env
executables:
  - label: python
    instances:
      - command: python
        num_cores: 1
        parallel_mode: null

damask#

Example environment definition#

name: damask_env
executables:
  - label: damask_grid
    instances:
      - command: docker run --rm --interactive --volume ${PWD}:/wd --env OMP_NUM_THREADS=1 eisenforschung/damask-grid:3.0.0-alpha7
        parallel_mode: null
        num_cores: 1