Creation and configuration of a meta-workflow on SSP

This tutorial shows you how to use the SSP and CGI technologies to create a meta-workflow and configure it with sub-workflows deployed on the SHIWA Repository.


Please give your feedback about the SHIWA project after you finish this tutorial: http://www.shiwa-workflow.eu/feedback Thank you!


Hands on application use case

This tutorial uses the LINGA Application as its use case.

LINGA aims to elaborate on an existing set of neuroscientific application toolkits (primarily focused on Alzheimer's Disease) in order to specify a "meta workflow” involving the synchronized and complementary use of CBRAIN, LONI and neuGRID distributed computing e-infrastructures resources. LINGA intends to demonstrate the benefits of interoperability across continental facilities, through the resolution of a “1-variable” problem materializing in the execution of the CIVET cortical thickness extraction pipeline applied to 3 different but compatible imaging data sets, each of which being hosted by respective e-infrastructure.

The meta-workflow of the LINGA application is composed of three workflow nodes which embed three CIVETs running on three different infrastructures (i.e. CBRAIN in Canada, LONI in the US and neuGRID in Europe), two synchronizing nodes which synchronize the execution of the three CIVETs and launch the execution of the quality control process.

Main steps of the tutorial


Step 1: Connect to SSP

A list of user accounts have been created with login/password template: testXY/shiwatestXY, where XY is a number between 01-15.

If you do not have yet an account on SSP, use one of accounts above to login to SSP to login (click on Sign In top-right corner).


Step 2: Download certificate and associate to SHIWA Grid

Click on Security -> Certificates menu, you will have interface where you manage your certificate.

If your certificate is not valid or there is not any certificate. Click on the Download button and then enter information about myProxy server to download a new certificate.

When your certificate is successfully downloaded. You should associate your certificate to GEMLCA Service. Click on Associate to VO

Choose gemlca-devel.cpc.wmin.ac.uk:8443 and then click OK


Step 3: Create a meta-graph corresponding to the meta-workflow using WS-PGRADE Portlet

Click on Workflow->Graph menu and then click on the Graph Editor to download WS-PGRADE Portlet to your local machine. When you successfully downloaded the portlet, open it on your machine to have the Graph Editor Interface.

On the graph editor, we will start now designing the LINGA Application Graph.

Repeat these two steps to create two other nodes representing the CIVET running on LONI and CBRAIN.

The output of these three CIVETs are then merged into a unique file for the last step of quality control. We will create a node called Synchronization that waits for all three CIVETs outputs and then merges into a file. It has 3 input ports and one output port as shown in the following figure.

After creating the Synchronization node, link the output ports of three CIVETs to its input ports.

You will have the resulting graph as shown in the following figure:

The last step to finalize the meta-graph is to create the Quality Control node. It also has one input port and one output port. Create this node and link the output port of Synchronization to the input port of this node. The final graph you will have as follow:

Save the meta-graph as LINGA-Meta-Graph before closing the portlet (Graph -> Save). Your meta-graph is stored on the SSP and appears in the list of graphs.


Step 4: Create a conrete meta-workflow based on meta-graph

Come back to the SSP Portal, you will now create a concrete meta-workflow based on the graph that you created in previous step. Use the menu Workflow -> Create Concrete, choose the graph you would like to use (LINGA-Meta-Graph in our case) and enter the name of concrete meta-workflow (for example: LINGA-Meta-Workflow). Finally, click OK to save the meta-workflow.

Your workflow will appear in the list of concrete workflows (Click on Workflow -> Concrete)


Step 5: Configure sub-workflows for the meta-workflow

On the list of concrete workflows, click on the Configure button of the meta-workflow that you just saved. You will have the configuration interface as following figure (the meta-graph is shown in the windows Graph of the workflow to be configured). You will then click on each node to configure and set the sub-workflow properties.

Configuration of CIVET-neuGRID, CIVET-LONI and CIVET-CBRAIN

Click on the CIVET-neuGRID node, you will have the its properties table shown in the windows Selected job to be configured. You should then configure all properties of Job Executable and Job I/O Tabs.

After finishing the configuration of the CIVET-neuGRID, you repeate these three steps above to configure CIVET-LONI and CIVET-CBRAIN nodes with different input file and service method:

Configuration of Synchronization

On the Graph of the workflow to be configured windows:

Configuration of Quality Control

Click on Quality Control node to configure its properties.


Step 6: Launch the execution of the meta-workflow

Click on Workflow -> Concrete to come back to the list of concrete workflows, you then click on Submit button of the meta-workflow LINGA-Meta-Workflow to launch its execution. To monitor its execution, click then on Details button. You will have a figure like:

You just finish the tutorial, please give your feedback about the SHIWA project: http://www.shiwa-workflow.eu/feedback Thank you!