airavata-architecture mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary E. Gorbet" <gegor...@gmail.com>
Subject Re: Airavata 0.16 Design Refactoring Suggestion?
Date Mon, 01 Jun 2015 15:20:40 GMT
I would just like to chime in that I agree with Amila’s general point that having multiple
logs capture everything and the Airavata APIs able to return a specified one of them (or set
of them) sounds like the most useful model.

Gary

> On Jun 1, 2015, at 9:11 AM, Amila Jayasekara <thejaka.amila@gmail.com> wrote:
> 
> Hi Shameera,
> 
> I am also having little difficulty relating the problem you are stating and
> the solution approach you are proposing. This is maybe because I am
> outdated with Airavata developments.
> 
> As per my understanding (from your description) "transparency" is the
> problem. One question about the problem is what level of transparency are
> you expecting ? Is it transparency depicted in log files or feedback to
> user front-ends etc. You may need to clarify this.
> 
> Also if it is the transparency in logs or user front-ends, then the
> solution you are proposing doesn't fit exactly to the problem (to my
> understanding).
> 
> On the other hand task based approach is a nice way to handle jobs. I guess
> you need to design a schedular to execute these tasks also. Some things you
> need to be concerned about when designing such a schedular are as follows;
> 
> 1. Some tasks have dependencies and ordering. Probably dependencies and
> ordering is defined by user (just like they way handlers are defined in
> gfac config files)
> 2. Would be nice to execute tasks in parallel whenever they dont have
> dependencies
> 3. Also be concerned about FT aspects when designing such a schedular.
> State of the schedular and also state of the individual tasks may need to
> be replicated over all nodes.
> 
> Thanks
> -Thejaka Amila
> 
> 
> 
> 
> On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <swsachith@gmail.com> wrote:
> 
>> Hi Shameera,
>> 
>> Wouldn't a job specific distributed logging mechanism solve the problem
>> that you mentioned? Allow the user/GWDeveloper to specify which
>> logs/messages
>> they want to see ( apart from that log everything anyways).
>> 
>> 
>> 
>> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com
>>> wrote:
>> 
>>> Hi Gary & Borries,
>>> 
>>> It seems architecture mailing list doesn't show(drop) attachments, here
>> is
>>> the link
>>> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
>>> 
>>> @Suresh, Could you please have a look ?
>>> 
>>> Thanks,
>>> Shameera.
>>> 
>>> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
>>> demeler@biochem.uthscsa.edu> wrote:
>>> 
>>>> I think your attachments are stripped somewhere, you would be better
>> off
>>>> posting a URL.
>>>> 
>>>> Thanks, -b.
>>>> 
>>>>> Hi Gary,
>>>>> 
>>>>> I converted the image to pdf and attached here. in both mails i can
>> see
>>>>> the
>>>>> image, wonder how you all not getting that image. Please let me know
>> if
>>>>> you
>>>>> still can't see it.
>>>>> 
>>>>> Thanks,
>>>>> Shameera.
>>>>> 
>>>>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <gegorbet@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>>> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
>>>>>> <shameerainfo@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Gary,
>>>>>>> 
>>>>>>> I inserted the diagram as image, let me attache it as attachment.
>>>>>> 
>>>>>> I still see nothing.
>>>>>> - Gary
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Shameera.
>>>>>>> 
>>>>>>> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <
>> gegorbet@gmail.com
>>>> 
>>>>>> wrote:
>>>>>>> Shameera,
>>>>>>> 
>>>>>>> On my copy of this email there was no attachment, no following
>>>>>> diagram.
>>>>>> Would you please send that attachment or a URL that points to it.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Gary
>>>>>>> 
>>>>>>>> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
>>>>>> <shameera@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Devs,
>>>>>>>> 
>>>>>>>> As we are about to release Airavata 0.15( already cut the
>> branch )
>>>>>> we
>>>>>> will not add any major changes and it is in testing stage. This will
>>>>>> give
>>>>>> us time to discuss and finalize requirements for the next release
,
>> it
>>>>>> can
>>>>>> be either 0.16 or 1.0.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> As per the feedback from our user community, they need more
>>>>>> transparent view of what Airavata does when they submit an
>> experiment
>>> to
>>>>>> run a job on remote computer resource. Airavata users are science
>>>>>> gateway
>>>>>> developers, they are not only interested in Experiment level and
>>> remote
>>>>>> Job
>>>>>> level status changes. They would like to know some degree of
>>>>>> transparency
>>>>>> about pre-processing and post-processing tasks performed by airavata
>>>>>> framework, before and after Job submission. For example they would
>>> like
>>>>>> to
>>>>>> see which task is being executed at particular time, does scp file
>>>>>> transferring succeed or not. With current Hander architecture, it
is
>>> not
>>>>>> possible to Airavata framework to know which handler does what. User
>>> can
>>>>>> write and integrate different kind of handlers and integrate it with
>>> the
>>>>>> execution chain. If Airavata Job submission failed while
>> transferring
>>>>>> input
>>>>>> file to the compute resource. Gateway developer should be able to
>> find
>>>>>> the
>>>>>> reason without any trouble. Current Airavata save the failure reason
>>>>>> with
>>>>>> stracktrace but that is too low level for a gateway developer.
>>>>>>>> 
>>>>>>>> Here we are thinking of replace this static handler architecture
>>>>>> with
>>>>>> dynamic task mechanism. Here framework has different type of tasks,
>>> lets
>>>>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each
>> task
>>>>>> clearly know what it need to do and how. When Airavata get an
>>> experiment
>>>>>> with three inputs, one is simple string and other two are SCP and
>> HTTP
>>>>>> type
>>>>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
>>> tasks
>>>>>> to
>>>>>> the dynamic task chain. Then add another Job submission task, let's
>>> say
>>>>>> job
>>>>>> need to submit using ssh keys then Airavata add SSH job submission
>>> task.
>>>>>> as
>>>>>> same add required task for the outputs. Each task has three states
>>>>>> Processing, Completed, Failed. In case of failure, framework know
>>> which
>>>>>> type of works it was doing or which task failed, is it SCP file
>>> transfer
>>>>>> task or GRIDFTP file transfering task. Then Airavata can
>> provide(show)
>>>>>> this
>>>>>> details to Users by messaging. Please see following diagram to get
>> an
>>>>>> idea
>>>>>> about different level of state transitions.
>>>>>>>> 
>>>>>>>> Yours feedback are highly appreciate. ​
>>>>>>>> 
>>>>>>>> ​
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Shameera.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Shameera Rathnayaka.
>>>>>>> 
>>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Best Regards,
>>>>> Shameera Rathnayaka.
>>>>> 
>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Borries Demeler, Ph.D.
>>>> Associate Professor
>>>> The University of Texas Health Science Center at San Antonio
>>>> Dept. of Biochemistry
>>>> Email: demeler@biochem.uthscsa.edu
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>> 
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>> 
>> 
>> 
>> 
>> --
>> Thanks,
>> Sachith Withana
>> 


Mime
View raw message