kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-360) Add ability to disable rebalancing in ZooKeeper consumer
Date Fri, 01 Jun 2012 21:15:23 GMT
Chris Riccomini created KAFKA-360:

             Summary: Add ability to disable rebalancing in ZooKeeper consumer
                 Key: KAFKA-360
                 URL: https://issues.apache.org/jira/browse/KAFKA-360
             Project: Kafka
          Issue Type: Bug
          Components: core
            Reporter: Chris Riccomini

There is a need for a ZooKeeper-based Kafka consumer that does not re-balance. This is needed
because we may be handling partitioning outside of Kafka. For example, I may have a stateful
process that is meant to consume only from Partition 7 of a given Kafka topic. When that process
stops, I don't want another consumer to take over partition 7.

The benefits we get from using the ZooKeeper-based consumer (vs the Simple Consumer) without
rebalancing is that offsets will still be handled by Kafka/ZK, as will failover when a partition's
leader disappears/fails.

I think the way to do this is to add a consumer config parameter that disables a consumer
group's rebalancing. That way, the first consumer in the group to connect (when the ZK node
is created) can specify if rebalancing should be enabled for the consumer group. If rebalancing
is disabled, the consumers should be forced to supply a list of partition IDs that they wish
to read from. Perhaps this can be done during the createMessageStreams call?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message