airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Current Input/Output Files Management Status
Date Mon, 20 Apr 2015 14:45:11 GMT
On Apr 20, 2015, at 7:04 AM, SmashRod Alfredo <smashrod@hotmail.it> wrote:
> 
> Hello Everyone,
> I've successfully compiled airavata 0.14 release on my machine launching the Echo sample
and creating another easy Test application (using the LS command).

Hi Alfredo,

Thank you for your persistent interest in Airavata, we appreciate it and are here to help
you. Please see below for responses: 

> Both experiments complete successfully and I've found the stdout result in the dedicated
experiment directory while Input and Output directories are empty.
> 
> I have some questions about possible Input/Output file management and the construction
of a little complex workflow:
> 1) In which case the Input/Output directories of a certain experiment will contain certain
files? I've try to define the output my "LS" application of type URI but nothing has been
generated. The physical generation of possible output files should be handled directly into
the application deployment?

Yes you are right and you stated it correctly in your assumptions below. If you rather not
want to have wrappers, you could be more expressive on how you want Airavata to detect the
outputs. Determining outputs in a general fashion is tricky and we typically go by the usecases.
If you can describe your example in little more detail, we can either suggest or improve the
output handling based on it.

> 2) How can I create a "complex" workflow which involves files on XBaya? I describe a
possible example:
>  The path to a certain folder on the local machine is the only input for the workflow.
>  This path is the input of a certain "Application1", which execution implies the generation
of a certain output file
>  Once Application1 processing is completed both the input path and the output file should
represent the "inputs" of a second "Application2" which execute some other processing and
generates other output files.
> 
> My assumption is the following:
> - Both Applications Deployments (which are wrappers (.sh) of external application) should
internally handle the creation and generation of the output files
> - In order to provide as input to the "Application2" interface the output file generated
by the "Application1" is necessary to define the output file as an another "Input" Element
in airavata XBaya workflow
> 
> Let me know if there are some other (and smarter) ways to create this workflow.

Yes what you suggest is certainly a way to go. If you want to build on it and have more advanced
scenarios, Airavata can handle remote file transfer protocols. So you can configure so that
Airavata can stage the locally generated output to a remote server which either has a SSH
access or GridFTP (which is more complex to setup, but we can guide if you are interested).
Adding WebDav but we never had a driving usecase. 

> 
> 3) How can I handle possible output file trasfer from a remote machine to the local one
executing the workflow?

Are the outputs in a public location accessible by http or ftp? We have to revisit all the
supported protocols, but in theory Airavata should be able to handle decent number of them.
For instance, if they are on a remote machine accessible by SSH/SCP, you can register ssh
keys with Airavata and it can secured move data to a local location. 

> 4) On airavata XBaya on deployed application interfaces are displayed (and connectable)
two special red circle icon (top-right and bottom-left) which seems to be a sort of possible
"Execution Order" directives. Is it that correct? If it's not can someone please explain what
they represent?

Sorry I did not try this myself. May be Shameera can explain this one. 

Suresh

> 
> I hope to have been sufficiently clear, if I didn't please tell me and I'll try to describe
better the questions.
> 
> PS: In January (13 January) there was a message of Raminder Singh reviewing the status
of output application handlers, there was some improvement or modification of the State of
Art of output handlers/management?
> 
> Thank you all
> 
> Alfredo


Mime
View raw message