apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject Re: Shutdown of an Apex app
Date Wed, 18 Jan 2017 07:05:24 GMT
Yes Ajay, for a graceful shutdown, the data sent out should be processed.

On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <ajaygit158@gmail.com> wrote:

> +1 to idea.
>
> Will this ensure downstream operators to process all data received before
> shutdown is called?
> Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
> operator somewhere downstream, and an operator in one of the sub-DAGs sends
> ShutdownException.
>
>
> Ajay
>
> On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
> wrote:
>
> > This JIRA is to stop the DAG in a crude manner, based on an error
> > condition. I think this might also need similar functionality as an error
> > condition can occur anywhere in the DAG.
> >
> > Perhaps we can modify the same JIRA to include a graceful + ungraceful
> > (kill) shutdown from any operator in the DAG.
> >
> > ~ Bhupesh
> >
> > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tushar@datatorrent.com>
> > wrote:
> >
> > > I think this would be a great addition for batch use cases or use
> > > cases were DAG needs to be shutdown after detecting some
> > > completion/error condition through the operator. We have one Jira
> > > Opened for such functionality
> > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > <bhupesh@datatorrent.com> wrote:
> > > > Hi All,
> > > >
> > > > Currently we can shutdown an Apex app in the following ways:
> > > > 1. Throw ShutdownException() from *all* the input operators
> > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > >
> > > > I think we should have some way of shutting down an application from
> > > within
> > > > an operator. It is not always true that the trigger for shutdown is
> > sent
> > > by
> > > > the input operator only. Sometimes, an end condition may be detected
> by
> > > > some operator in the DAG which wants the processing to end. Such a
> > > > shutdown, although triggered from some intermediate operator in the
> > DAG,
> > > > should guarantee graceful shut down of the application.
> > > >
> > > > Thoughts?
> > > >
> > > > ~ Bhupesh
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message