airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SmashRod Alfredo <>
Subject RE: Current Input/Output Files Management Status
Date Wed, 22 Apr 2015 06:27:47 GMT
Thanks for your reply Suresh,
so basically to manage Output values there are 2 ways:
-> Create an application directly from the airavata API to better handle output value generation
-> (Trick way) have an application that on stdout streams the key-value couple <OutputName>=<OutputValue>

Does Airavata made any improvements for structured input/output? (Custom definition of structure
starting from the basic String/URI/boolean etc)?

I have two others general question but I'm going to create appropriate thread for that.

Thanks again


Subject: Re: Current Input/Output Files Management Status
Date: Mon, 20 Apr 2015 10:45:11 -0400

On Apr 20, 2015, at 7:04 AM, SmashRod Alfredo <> wrote:
Hello Everyone,
I've successfully compiled airavata 0.14 release on my machine launching the Echo sample and
creating another easy Test application (using the LS command).

Hi Alfredo,
Thank you for your persistent interest in Airavata, we appreciate it and are here to help
you. Please see below for responses: 
Both experiments complete successfully and I've found the stdout result in the dedicated experiment
directory while Input and Output directories are empty.

I have some questions about possible Input/Output file management and the construction of
a little complex workflow:
1) In which case the Input/Output directories of a certain experiment will contain certain
files? I've try to define the output my "LS" application of type URI but nothing has been
generated. The physical generation of possible output files should be handled directly into
the application deployment?

Yes you are right and you stated it correctly in your assumptions below. If you rather not
want to have wrappers, you could be more expressive on how you want Airavata to detect the
outputs. Determining outputs in a general fashion is tricky and we typically go by the usecases.
If you can describe your example in little more detail, we can either suggest or improve the
output handling based on it.
2) How can I create a "complex" workflow which involves files on XBaya? I describe a possible
 The path to a certain folder on the local machine is the only input for the workflow.
 This path is the input of a certain "Application1", which execution implies the generation
of a certain output file
 Once Application1 processing is completed both the input path and the output file should
represent the "inputs" of a second "Application2" which execute some other processing and
generates other output files.

My assumption is the following:
- Both Applications Deployments (which are wrappers (.sh) of external application) should
internally handle the creation and generation of the output files
- In order to provide as input to the "Application2" interface the output file generated by
the "Application1" is necessary to define the output file as an another "Input" Element in
airavata XBaya workflow

Let me know if there are some other (and smarter) ways to create this workflow.

Yes what you suggest is certainly a way to go. If you want to build on it and have more advanced
scenarios, Airavata can handle remote file transfer protocols. So you can configure so that
Airavata can stage the locally generated output to a remote server which either has a SSH
access or GridFTP (which is more complex to setup, but we can guide if you are interested).
Adding WebDav but we never had a driving usecase. 

3) How can I handle possible output file trasfer from a remote machine to the local one executing
the workflow?

Are the outputs in a public location accessible by http or ftp? We have to revisit all the
supported protocols, but in theory Airavata should be able to handle decent number of them.
For instance, if they are on a remote machine accessible by SSH/SCP, you can register ssh
keys with Airavata and it can secured move data to a local location. 
4) On airavata XBaya on deployed application interfaces are displayed (and connectable) two
special red circle icon (top-right and bottom-left) which seems to be a sort of possible "Execution
Order" directives. Is it that correct? If it's not can someone please explain what they represent?

Sorry I did not try this myself. May be Shameera can explain this one. 

I hope to have been sufficiently clear, if I didn't please tell me and I'll try to describe
better the questions.

PS: In January (13 January) there was a message of Raminder Singh reviewing the status of
output application handlers, there was some improvement or modification of the State of Art
of output handlers/management?

Thank you all

View raw message