airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raminder Singh <raminderjsi...@gmail.com>
Subject Re: Re-architecture work in gfac-core
Date Wed, 30 Jan 2013 17:17:14 GMT
Please see my comments inline. 

On Jan 30, 2013, at 11:14 AM, Lahiru Gunathilake wrote:

> Hi Raman,
> 
> On Wed, Jan 30, 2013 at 10:49 AM, Raminder Singh
> <raminderjsingh@gmail.com>wrote:
> 
>> Hi Lahiru,
>> 
>> Good you guys are working on re-architecting GFAC. I have following
>> requirements from different projects
>> 
>> 1. Job status/listener recovery in case GFAC or other airavata component
>> need a restart. Currently all the job listeners stop and i need to manually
>> pull the results. This is the case with workflow also. User don't want to
>> start the workflow from the start in case of Airavata failures.
>> 
> Can you pleas explain bit more about this ?
Use case1 : There can be few long running jobs in Airavata in running state and we are forced
to restart Airavata server for some reason (memory error/ bug fix etc). Now the current status
of those jobs are lost and someone need to manually monitor the jobs and get the results.
It will be really nice if GFAC can save job state and restart the listener from the current
state the job. In case job finished while Airavata server is down then GFAC can complete post
processing steps.
Use case 2: In case GFAC is able to save the state of the job and recover from that state,
same challenge is with the workflow. I can be at step 3 of the workflow when Airavata server
went down. User will want to proceed from there after Airavata server comes up. We may need
to save state of the workflow also and recover from there. 

These features need to be configurable using some property for the development purpose as
developers may not want to recover the workflow. 

Please let me know if you want me to explain more. 

> 
>> 2. Job cancel incase user feels job was not configured right.
>> 
> This features anyhow wasn't there in old gfac and the reason was there's no
> direct support from gram, so we can think of a new solution for this once
> we have the new gfac working.
Gram API have cancel support but we are not managing any state in GFAC so we cant cancel the
job. I implemented a solution for ultrascan to save the Gram jobid while submitting the job
and cancel the job by creating new object of Gram Job with same jobid. Doing that GFAC listener
terminates with job status as cancel. We can have a Airavata API job cancel function but we
need to find how to get all the job id's of active jobs and how to terminate the workflow
in case of cancel. 
> 
>> 3. Provide interface to plugin any other messing system like RabbitMQ. SQL
>> or NoSQL DB can also be used to save messages.
>> 
> This we are going to support, allowing users to have its own notifier
> configured.
> 
>> 4. Working of Input and output chain. I have to upload the job results to
>> a database after the job finish. That is done using output extension in
>> previous version of GFAC.
>> 
> New architecture supports this.
> 
>> 5. Gram job retry incase of certain error codes.
>> 
> Noted.
> 
> Lahiru
> 
>> 
>> Please let me know if i can help with anything.
>> 
>> Thanks
>> Raminder
>> 
>> 
>> On Jan 28, 2013, at 10:10 PM, Suresh Marru wrote:
>> 
>>> Hi Lahiru & Milinda,
>>> 
>>> Good to hear the GSOC project contributions are making into trunk. Can
>> we please arrange for a virtual hackethon and discuss the architecture and
>> share the minutes to the mailing lists and iterate on them until we arrive
>> at a decision on the architecture changes?
>>> 
>>> A while ago I create a google plus community [1], please feel to add
>> yourself so we have a persistent hangout channel. (the community is yet to
>> be verifies, after it is done, we should get a #airavata url).
>>> 
>>> Suresh
>>> 
>>> [1] - https://plus.google.com/u/0/communities/100700433662281905708
>>> 
>>> On Jan 28, 2013, at 5:12 PM, Lahiru Gunathilake <glahiru@gmail.com>
>> wrote:
>>> 
>>>> Hi Devs,
>>>> 
>>>> Me and Milinda are going to port his GSoc work to airavata-trunk, so if
>>>> anyone is working on gfac-core, gfac-schema,workflow-tracking please do
>>>> small commits, so that we can copy that code to our branch time to time
>> and
>>>> sync with trunk. If you do huge commits to trunk things will be tough to
>>>> merge with re-architecture work.
>>>> 
>>>> Thanks
>>>> Lahiru
>>>> 
>>>> --
>>>> System Analyst Programmer
>>>> PTI Lab
>>>> Indiana University
>>> 
>> 
>> 
> 
> 
> -- 
> System Analyst Programmer
> PTI Lab
> Indiana University


Mime
View raw message