kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sophie Blee-Goldman <sop...@confluent.io>
Subject Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown
Date Fri, 11 Sep 2020 22:39:57 GMT
Hey Walker,

The proposal makes sense to me, but while reading up on those old
tickets I started wondering if we should give users two options: one
that would shut down the entire application, as described in the current
KIP, and another that would only shut down an individual instance.

I think there are a number of reasons that shutting down only the
erroring instance and not the entire application would be useful, for
example if the local state is corrupted. Currently a user can achieve
this, but the mechanism is pretty awkward and has caused issues in
the past: calling close inside the uncaught exception handler. The uncaught
exception handler is called from the stream thread, and close(timeout) is
supposed to wait for all stream threads to shutdown. So you end up waiting
for the full timeout to pass and even once it does, you know for a fact that
not all threads have completed shutdown. Not to mention, if you happen
to call the close() overload with no timeout parameter, your application
will be stuck blocking forever.

Giving users an exception to throw that means "shutdown the instance"
would make this common practice significantly smoother. It shouldn't
expand the scope of this KIP too much since we'll have a mechanism to
shut down the instance already in place.  WDYT?

On that note, it might be a good idea to choose a more explicit name for
the exception in the current proposal, eg `ShutdownApplicationException`,
so it's obvious what exactly will shut down. Then we could have the second
exception be named `ShutdownInstanceException` or so on


On Fri, Sep 11, 2020 at 11:08 AM Matthias J. Sax <mjsax@apache.org> wrote:

> Thanks for the KIP.
> It seem that the new exception would need to be thrown by user code?
> However, in the motivation you mention the scenario of a missing source
> topic that a user cannot detect, but KafkaStreams runtime would be
> responsible to handle.
> How do both things go together?
> -Matthias
> On 9/11/20 10:31 AM, Walker Carlson wrote:
> > Hello all,
> >
> > I have created KIP-671 to give the option to shutdown a streams
> > application in response to an error.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-671%3A+Shutdown+Streams+Application+when+appropriate+exception+is+thrown
> >
> > This is because of the Jira ticket
> > <https://issues.apache.org/jira/browse/KAFKA-9331>
> >
> > Please give it a look and let me know if you have any feedback.
> >
> > Thanks,
> > Walker
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message