apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: Shutdown of an Apex app
Date Mon, 23 Jan 2017 12:54:45 GMT
Hi Thomas,

Handling application shutdown from an operator with help of stats
listener can be lot of work, instead we could provide an shortcut api
for
user to terminate the application. It could be based on response code
from statsListener as well as throwing an special exception from
operator thread or setting a special request through Context.

As far as I know the statsListener approach would involve
- Adding a shared stats listener between operator detecting
termination condition and with all input operators.
- stats listener sending an OperatorRequest whilch will throw
ShutdowException in execute method.

- Tushar.


On Wed, Jan 18, 2017 at 10:35 PM, Thomas Weise <thw@apache.org> wrote:
> Bhupesh,
>
> If your intention is to let processing complete, the sources should
> terminate the stream (that's already implemented, except that you want to
> initiate it from another operator). It does not really matter if they all
> stop at the same streaming window.
>
> If you want to just stop the application from an operator (similar to
> kill), it needs to be signaled to the master and the master needs to
> terminate. One way of doing it could be the stats listener.
>
> Thomas
>
>
> On Tue, Jan 17, 2017 at 10:30 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
> wrote:
>
>> This JIRA is to stop the DAG in a crude manner, based on an error
>> condition. I think this might also need similar functionality as an error
>> condition can occur anywhere in the DAG.
>>
>> Perhaps we can modify the same JIRA to include a graceful + ungraceful
>> (kill) shutdown from any operator in the DAG.
>>
>> ~ Bhupesh
>>
>> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tushar@datatorrent.com>
>> wrote:
>>
>> > I think this would be a great addition for batch use cases or use
>> > cases were DAG needs to be shutdown after detecting some
>> > completion/error condition through the operator. We have one Jira
>> > Opened for such functionality
>> > https://issues.apache.org/jira/browse/APEXCORE-503.
>> >
>> > - Tushar.
>> >
>> >
>> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
>> > <bhupesh@datatorrent.com> wrote:
>> > > Hi All,
>> > >
>> > > Currently we can shutdown an Apex app in the following ways:
>> > > 1. Throw ShutdownException() from *all* the input operators
>> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
>> > >
>> > > I think we should have some way of shutting down an application from
>> > within
>> > > an operator. It is not always true that the trigger for shutdown is
>> sent
>> > by
>> > > the input operator only. Sometimes, an end condition may be detected by
>> > > some operator in the DAG which wants the processing to end. Such a
>> > > shutdown, although triggered from some intermediate operator in the
>> DAG,
>> > > should guarantee graceful shut down of the application.
>> > >
>> > > Thoughts?
>> > >
>> > > ~ Bhupesh
>> >
>>

Mime
View raw message