Table of Contents
Soma-workflow is a unified and simple interface to parallel computing resources. It is an open source Python application which aims at making easier the use of parallel resources by non expert users and external software.
This tool is included in BrainVISA package since version 4.1.0 and is really integrated in BrainVISA graphical user interface since version 4.2.0. It can be used to easily distribute BrainVISA iterations on a multi-core machine or on a cluster.
A BrainVISA pipeline or iteration, associated to a set of parameters, can be converted to an execution workflow. An execution workflow is simply a set of jobs, mainly defined by a program command line (a call to BrainVISA in batch mode in this context) and the execution dependencies between the jobs. Soma-workflow distributes the execution of workflows on the computing resource of your choice, including your multiple core machine.
The documentation of the Soma-workflow software is available here. In this pages, you will particularly find more details about the graphical user interface and the execution of workflows on a cluster.
Warning
For the moment, the parallel computing with Soma-workflow is not available in basic user level. To use it, you will first have to change the user level to Advanced or Expert in BrainVISA preferences.Using Soma-workflow in BrainVISA to distribute processes execution on a multi-core machine is very easy and does not need any particular configuration steps. The execution can be really faster if the machine has several processors. Indeed, by default BrainVISA uses only one processor.
Here are the steps to distribute a BrainVISA iteration on the processors of your computer:
Iterate a process or a pipeline on a set of data. In the iteration window, a button Run in parallel should be available in addition of the usual Run button.
Click on the Run in parallel button. A window Workflow submission appears. You can let the default values for the parameters except the workflow name, you can type a custom name to easily retrieve this workflow later.
Click on the Ok button in the workflow submission window. At this step an execution workflow is generated and starts to run. A new window appears to enable the monitoring of the workflow execution. You can stop and restart the execution at anytime using the Stop and Restart buttons in the toolbar. Since this window contains elements of the Soma-workflow graphical interface, we invite you to refer to Soma-workflow documentation for a detailed description.
It is possible to display the corresponding BrainVISA process in the same window by clicking on the
icon in the toolbar at the top of the window. To guarantee that the parameters displayed here always correspond to the execution workflow displayed on the right, the process is in read-only mode: it is not possible to change the parameters. If you want to modify the parameters, you have to open a copy of the process using the Edit button and run it in parallel as a new workflow which corresponds to the new set of parameters.
When you are monitoring a workflow execution, the various colors indicates the status of the workflow subparts. Blue means "running", green means "ended with success" and red means "failed". In case a subpart fails you can expand it (clicking on the small cross) to display the jobs which are inside. The job standard output and error, displayed at the bottom of the window, will help you to work out what happened.
The window which displays the workflow execution can be closed at any time. However, if you are using your machine to execute your workflows (in opposition to a remote cluster for example) you should not close BrainVISA before the end of the execution. A summary of the status of the workflows is displayed at any time in the main window of BrainVISA in the "Execution" panel (use the menu View -> Workflow execution to get it visible). In the same panel, the Add button can be used to connect to other computing resources (a remote cluster for example) on which you will be able to submit and monitor your workflows the same way (see documentation about Soma-workflow client-server application configuration here).
You can double click on the "Execution" panel to display a list of the current workflows and double click on a workflow to display details about this particular workflow.
Warning
Soma-workflow runs BrainVISA in a batch mode to execute each job and in this mode, some features linked to BrainVISA databases are disabled. So you may need to check and update your databases after a parallel run with Soma-Workflow. To do so, use the menu Process -> Check & update databases.
Another way to do the same is to run Data management -> Update databases process and then Data management -> Convert old database steps -> 2 - Check database.