airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amila Jayasekara <thejaka.am...@gmail.com>
Subject Re: Airavata 0.16 Design Refactoring Suggestion?
Date Wed, 03 Jun 2015 02:28:11 GMT
Hi Shameera,

By looking at the diagram I can ask lot of questions but I guess your
complete design is not depicted in the picture.

Few quick questions are as follows;

1. What is API server ? (sorry I dont know what this is) and what is the
purpose of that ?
2. Is experiment id generated by the PHP gateway (as per diagram that is
the idea i got) ? (i.e. by client side not Airavata server) If so I would
like to know how it works.
3. When you first mention about tasks I was under the impression that GFac
is going to spawn different tasks for a process. i.e. Scp task, Qsub task
etc. but diagram doesnt exactly depict that behaviour.
4. Is monitoring also a task ?
5. Seems some tasks need to share information among other components (or
other tasks). E.g:- job id from job submission task. How are you planning
to share those data ?
6. Also diagram doesnt show how this design resolves your original question
("transparency"); i.e. sending feedback about each task to amqp
7. Also did you consider FT in the design ? It seems tasks have a pretty
important role to acheive FT. E.g:- The job submission task

I remember we discussing a design on FT sometime back and I believe Saminda
came up with few nice diagrams that handle most of the failure cases. I
suggest you looking at them if you are planning to consider FT in your
design.

Thanks
-Thejaka


On Tue, Jun 2, 2015 at 6:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com>
wrote:

> Hi All,
>
> Here is how the sequence diagram looks like for experiment submit request,
> with the proposed changes.
>
>
>
> ​
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <shameera@apache.org>
> wrote:
>
>> Hi Devs,
>>
>> As we are about to release Airavata 0.15( already cut the branch ) we
>> will not add any major changes and it is in testing stage. This will give
>> us time to discuss and finalize requirements for the next release , it can
>> be either 0.16 or 1.0.
>>
>>
>> As per the feedback from our user community, they need more transparent
>> view of what Airavata does when they submit an experiment to run a job on
>> remote computer resource. Airavata users are science gateway developers,
>> they are not only interested in Experiment level and remote Job level
>> status changes. They would like to know some degree of transparency about
>> pre-processing and post-processing tasks performed by airavata framework,
>> before and after Job submission. For example they would like to see which
>> task is being executed at particular time, does scp file transferring
>> succeed or not. With current Hander architecture, it is not possible to
>> Airavata framework to know which handler does what. User can write and
>> integrate different kind of handlers and integrate it with the execution
>> chain. If Airavata Job submission failed while transferring input file to
>> the compute resource. Gateway developer should be able to find the reason
>> without any trouble. Current Airavata save the failure reason with
>> stracktrace but that is too low level for a gateway developer.
>>
>> Here we are thinking of replace this static handler architecture with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>> the dynamic task chain. Then add another Job submission task, let's say job
>> need to submit using ssh keys then Airavata add SSH job submission task. as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>> details to Users by messaging. Please see following diagram to get an idea
>> about different level of state transitions.
>>
>> Yours feedback are highly appreciate. ​
>>
>> ​
>>
>>
>> Thanks,
>> Shameera.
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Mime
View raw message