apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Shutdown of an Apex app
Date Wed, 18 Jan 2017 07:40:36 GMT
Can be done by sending "shutdown" message via heartbeat to Stram. Then on
stram can shutdown the entire app

Thks
Amol


On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> Yes Ajay, for a graceful shutdown, the data sent out should be processed.
>
> On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <ajaygit158@gmail.com> wrote:
>
> > +1 to idea.
> >
> > Will this ensure downstream operators to process all data received before
> > shutdown is called?
> > Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
> > operator somewhere downstream, and an operator in one of the sub-DAGs
> sends
> > ShutdownException.
> >
> >
> > Ajay
> >
> > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <
> bhupesh@datatorrent.com>
> > wrote:
> >
> > > This JIRA is to stop the DAG in a crude manner, based on an error
> > > condition. I think this might also need similar functionality as an
> error
> > > condition can occur anywhere in the DAG.
> > >
> > > Perhaps we can modify the same JIRA to include a graceful + ungraceful
> > > (kill) shutdown from any operator in the DAG.
> > >
> > > ~ Bhupesh
> > >
> > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > I think this would be a great addition for batch use cases or use
> > > > cases were DAG needs to be shutdown after detecting some
> > > > completion/error condition through the operator. We have one Jira
> > > > Opened for such functionality
> > > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > > <bhupesh@datatorrent.com> wrote:
> > > > > Hi All,
> > > > >
> > > > > Currently we can shutdown an Apex app in the following ways:
> > > > > 1. Throw ShutdownException() from *all* the input operators
> > > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > > >
> > > > > I think we should have some way of shutting down an application
> from
> > > > within
> > > > > an operator. It is not always true that the trigger for shutdown
is
> > > sent
> > > > by
> > > > > the input operator only. Sometimes, an end condition may be
> detected
> > by
> > > > > some operator in the DAG which wants the processing to end. Such
a
> > > > > shutdown, although triggered from some intermediate operator in the
> > > DAG,
> > > > > should guarantee graceful shut down of the application.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > ~ Bhupesh
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message