incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (Commented) (JIRA)" <>
Subject [jira] [Commented] (KAFKA-197) Embedded consumer doesn't shut down if the server can't start
Date Mon, 05 Dec 2011 05:37:40 GMT


Jay Kreps commented on KAFKA-197:

A less invasive way would just be to have the embedded consumer register a shutdown hook and
use System.exit.

I am a little concerned about this whole embedded consumer thing, though. The original approach
where we wrote to the local log in process was pretty fool proof. I think sending to a remote
broker is actually riddled with issues. The producer send buffer is vulnerable to quite a
large loss on any unclean shutdown or indeed any shutdown bugs. And also any condition that
leads to a broker being unable to take requests but still registered in zk will lead to unbounded
data loss. I wonder if this issue isn't just a special case of many many bad things that could

With the current approach I actually don't see any benefits at all to bundling the replication
process with the kafka broker. It would actually be better to have that run independently
it seems to me.
> Embedded consumer doesn't shut down if the server can't start
> -------------------------------------------------------------
>                 Key: KAFKA-197
>                 URL:
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.7
>            Reporter: Jun Rao
>             Fix For: 0.7.1
>         Attachments: KAFKA-197.patch
> If a broker embeds a consumer and the broker itself doesn't start (e.g., conflicting
broker id in ZK), the embedded consumer is still running. In this case, we should probably
shut down the embedded consumer too.
> To do this, we need to either throw an exception or return an error in KafkaServer.startup
and act accordingly in KafkaServerStartable.startup.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message