apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJAY GUPTA <ajaygit...@gmail.com>
Subject Re: Shutdown of an Apex app
Date Wed, 18 Jan 2017 06:49:38 GMT
+1 to idea.

Will this ensure downstream operators to process all data received before
shutdown is called?
Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
operator somewhere downstream, and an operator in one of the sub-DAGs sends
ShutdownException.


Ajay

On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> This JIRA is to stop the DAG in a crude manner, based on an error
> condition. I think this might also need similar functionality as an error
> condition can occur anywhere in the DAG.
>
> Perhaps we can modify the same JIRA to include a graceful + ungraceful
> (kill) shutdown from any operator in the DAG.
>
> ~ Bhupesh
>
> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
> > I think this would be a great addition for batch use cases or use
> > cases were DAG needs to be shutdown after detecting some
> > completion/error condition through the operator. We have one Jira
> > Opened for such functionality
> > https://issues.apache.org/jira/browse/APEXCORE-503.
> >
> > - Tushar.
> >
> >
> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > <bhupesh@datatorrent.com> wrote:
> > > Hi All,
> > >
> > > Currently we can shutdown an Apex app in the following ways:
> > > 1. Throw ShutdownException() from *all* the input operators
> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > >
> > > I think we should have some way of shutting down an application from
> > within
> > > an operator. It is not always true that the trigger for shutdown is
> sent
> > by
> > > the input operator only. Sometimes, an end condition may be detected by
> > > some operator in the DAG which wants the processing to end. Such a
> > > shutdown, although triggered from some intermediate operator in the
> DAG,
> > > should guarantee graceful shut down of the application.
> > >
> > > Thoughts?
> > >
> > > ~ Bhupesh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message