airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: Implementing the cancel/terminate
Date Wed, 30 Apr 2014 00:11:40 GMT
Hi Danushka,

I am thinking of having another operation for a handler which will be
invoked during a cancel process. If each handler wants to reset what it did
handlers can implement that logic to the same handler  class, if there
nothing much can be done handler can leave the cancel invocation blank.
Unless we do this its going to be hard to reset the operations did in each
handler.

We have handlers to run in daemon mode too, so for particular
implementation developers can use them to perform more higher level clean
up task which could not make sense to implement in each handler. So GFac
framework can provide handler level clean up but developers will not be
completely bound to use that for their clean up implementation.

Regards
Lahiru




On Tue, Apr 29, 2014 at 4:58 PM, Marlon Pierce <marpierc@iu.edu> wrote:

> I assumed there would be a lot of overlap between cancel-handlers and
> out-handlers but not complete overlap.  Cleanup operations need to be
> done in either case, but "move the successfully created output files to
> a safe storage location" would not be done.
>
> Marlon
>
> On 4/29/14 4:49 PM, Danushka Menikkumbura wrote:
> > To me this sounds a little more complicated.If we take the Whirr-based
> > cloud bursting impl for instance,we have an in handler to launch a
> cluster
> > and an out handler to shut it down.When a job is cancelled in a case like
> > this, the corresponding cluster should be shut down and for that the out
> > handler needs to be notified.
> >
> > /Danushka
> >
> > On Tuesday, April 29, 2014, Lahiru Gunathilake <glahiru@gmail.com>
> wrote:
> >> Hi Marlon,
> >>
> >> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <marpierc@iu.edu> wrote:
> >>> Are any handlers called when canceling? Should they be?  Seems like you
> >>> would need to typically do some cleanup.
> >> +1 thats the plan to have pre/post handlers for terminate operation like
> > we have to submit.
> >> Lahiru
> >>> Marlon
> >>>
> >>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
> >>>> Got it Suresh. Thanks for putting it clearly :)
> >>>>
> >>>>
> >>>> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <smarru@apache.org>
> > wrote:
> >>>>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <
> samindaw@gmail.com
> >> wrote:
> >>>>>> You mean like a monitoring client at client end triggering a
> > listener?
> >>>>> Thats ideal, but different story altogether. I mean triggering a
> > listener
> >>>>> for all major status changes including canceling.
> >>>>>
> >>>>> In this context I mean once the client gets Canceling at experiment
> > level,
> >>>>> once all lower level statues gets updated, we should ensure
> subsequent
> >>>>> client query to experiment returned Cancelled. I see your plan
> already
> >>>>> accounts for it, was just restating its simpler for gateways to
wait
> > on
> >>>>> experiment status instead of expecting to navigate through Task->Job
> > to
> >>>>> verify the success.
> >>>>>
> >>>>> Suresh
> >>>>>
> >>>>>
> >>>>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <smarru@apache.org>
> > wrote:
> >>>>>>> Hi Saminda,
> >>>>>>>
> >>>>>>> This like a good plan, I think we should make sure the API
server
> >>>>>>> properly reports these statuses back to the clients.
> >>>>>>>
> >>>>>>> Suresh
> >>>>>>>
> >>>>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <samindaw@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>  Hi All,
> >>>>>>>
> >>>>>>> After looking at the current design and doing some trial
and error
> I
> >>>>>>> thought of implementing the cancellation as follows.
> >>>>>>>
> >>>>>>>
> >>>>>>>    - Cancellation of an experiment requested by a gateway
requires
> >>>>>>>    cancellation request to go through several layers. (Orchestrator
> >> GFac >
> >>>>>>>    GFac Provider)
> >>>>>>>    - Each layer is responsible for handling cancellation
relevant
> > for
> >>>>>>>    that layer (Orchestration cancels experiment, GFac cancels
Task,
> > GFac
> >>>>>>>    Provider cancels Job)
> >>>>>>>    - What I thought is, each layer will listen to cancellation
> > request
> >>>>>>>    made to the layer above and perform its cancellation
actions
> > accordingly.
> >>>>>>>    (GFac will see the experiment is having the status "canceling"
> > for an
> >>>>>>>    experiment id and it will perform cancellation of the
tasks
> > relevant for
> >>>>>>>    that experiment)
> >>>>>>>       - Effectively the Orchestrator will be
> >>>>>>>       - updating the status of the experiment in registry
with the
> >>>>>>>          status "canceling"
> >>>>>>>          - publish a message which will be caught by GFac
instance
> >>>>>>>          which handles its Tasks.
> >>>>>>>          - GFac will perform the same and the correct GFac
Provider
> >>>>>>>       instance will catch the message and perform the actual
job
> > cancellation.
> >>>>>>>    - Once the job cancellation is done the statuses at each
layer
> > will
> >>>>>>>    be updated (to "canceled") in  similar fashion.
> >>>>>>>    - We allow the API call of cancellation to be asynchronous
> >>>>>>>    - I'm hoping to use the MonitorPublisher implemented
by Lahiru
> to
> >>>>>>>    publish the messages.
> >>>>>>>
> >>>>>>> wdyt?
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Saminda
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
>
>


-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
View raw message