kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5973) ShutdownableThread catching errors can lead to partial hard to diagnose broker failure
Date Thu, 05 Oct 2017 23:39:01 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193918#comment-16193918

Ted Yu commented on KAFKA-5973:

bq. which to kill itself, which to kill the whole process

In description, Tom listed threads which use {{ShutdownableThread}}. Is any of them not in
the above category ?

> ShutdownableThread catching errors can lead to partial hard to diagnose broker failure
> --------------------------------------------------------------------------------------
>                 Key: KAFKA-5973
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5973
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions:,
>            Reporter: Tom Crayford
>             Fix For:, 1.0.1
>         Attachments: 5973.v1.txt
> When any kafka broker {{ShutdownableThread}} subclasses crashes due to an
> uncaught exception, the broker is left running in a very weird/bad state with some
> threads not running, but potentially the broker can still be serving traffic to
> users but not performing its usual operations.
> This is problematic, because monitoring may say that "the broker is up and fine", but
in fact it is not healthy.
> At Heroku we've been mitigating this by monitoring all threads that "should" be
> running on a broker and alerting when a given thread isn't running for some
> reason.
> Things that use {{ShutdownableThread}} that can crash and leave a broker/the controller
in a bad state:
> - log cleaner
> - replica fetcher threads
> - controller to broker send threads
> - controller topic deletion threads
> - quota throttling reapers
> - io threads
> - network threads
> - group metadata management threads
> Some of these can have disasterous consequences, and nearly all of them crashing for
any reason is a cause for alert.
> But, users probably shouldn't have to know about all the internals of Kafka and run thread
dumps periodically as part of normal operations.
> There are a few potential options here:
> 1. On the crash of any {{ShutdownableThread}}, shutdown the whole broker process
> We could crash the whole broker when an individual thread dies. I think this is pretty
reasonable, it's better to have a very visible breakage than a very hard to detect one.
> 2. Add some healthcheck JMX bean to detect these thread crashes
> Users having to audit all of Kafka's source code on each new release and track a list
of "threads that should be running" is... pretty silly. We could instead expose a JMX bean
of some kind indicating threads that died due to uncaught exceptions
> 3. Do nothing, but add documentation around monitoring/logging that exposes this error
> These thread deaths *do* emit log lines, but it's not that clear or obvious to users
they need to monitor and alert on them. The project could add documentation

This message was sent by Atlassian JIRA

View raw message