kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean McKibben (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-4047) Return more useful information from ConsumerGroupCommand for consumer groups that are rebalancing or manually assigned
Date Tue, 16 Aug 2016 15:58:20 GMT
Sean McKibben created KAFKA-4047:
------------------------------------

             Summary: Return more useful information from ConsumerGroupCommand for consumer
groups that are rebalancing or manually assigned
                 Key: KAFKA-4047
                 URL: https://issues.apache.org/jira/browse/KAFKA-4047
             Project: Kafka
          Issue Type: Improvement
          Components: consumer, tools
    Affects Versions: 0.10.0.1, 0.10.0.0, 0.9.0.1
            Reporter: Sean McKibben
            Priority: Minor


For consumers which are manually assigned topic partitions, or for offsets which are otherwise
manually committed to brokers using the new consumer API, the existing ConsumerGroupCommand
tools are operationally insufficient.
There are many use cases in production operation when it is important to be able to easily
retrieve the stored offsets and calculated lag of a consumer group, even if the group is rebalancing
or partition assignment information is indeterminate. Often these cases involve a misbehaving
or crashed client application, and having the ConsumerGroupCommand return {{Consumer group
`myGroupID` does not exist or is rebalancing.}} instead of the information it does know is
not very helpful. Additionally, when manual offset commits are used, or the automatic consumer
group subscription management is not used, the same message is returned by the tool. In these
cases, the offsets are actually stored and available even though the broker/coordinator doesn't
have information about partition ownership.
Previously, this was a non-issue as a ZK client could be leveraged relatively easily to get
important information like the last stored offset positions, or consumer-offset-checker would
return more information in its responses. With the new consumer, however, the process for
getting offset and lag is not as straightforward, and tools to provide this information at
the operational level are necessary as part of a Kafka installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message