kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Gustafson (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-6287) Inconsistent protocol type for empty consumer groups
Date Wed, 24 Jan 2018 04:23:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Gustafson resolved KAFKA-6287.
    Resolution: Fixed

> Inconsistent protocol type for empty consumer groups
> ----------------------------------------------------
>                 Key: KAFKA-6287
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6287
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 1.0.0
>            Reporter: Ryan Leslie
>            Assignee: Jason Gustafson
>            Priority: Minor
>             Fix For: 1.0.1
> When a consumer is created for a new group, the group metadata's protocol type is set
to 'consumer' and this is stored both in __consumer_offsets as well as in the coordinator's
local cache.
> If the consumer leaves the group and the group becomes empty, ListGroups requests will
continue to show the group as type 'consumer', and as such kafka-consumer-groups.sh will show
it via --list.
> However, if the coordinator (broker) node is killed and a new coordinator is elected,
when the GroupMetadataManager loads the group from __consumer_offsets into its cache, it will
not set the protocolType if there are no active consumers. As a result, the group's protocolType
will now become the empty string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh will
no longer show the group.
> Ideally bouncing a broker should not result in the group's protocol changing. protocolType
can be set in GroupMetadataManager.readGroupMessageValue() to always reflect what's present
in the persistent metadata (__consumer_offsets) regardless if there are active members.
> In general, things can get confusing when distinguishing between 'consumer' and non-consumer
groups. For example, DescribeGroups and OffsetFetchRequest does not filter on protocol type,
so it's possible for kafka-consumer-groups.sh to describe groups (--describe) without actually
being able to list them. In the protocol guide, OffsetFetchRequest / OffsetCommitRequest have
their parameters listed as 'ConsumerGroup', but in reality these can be used for groups of
unknown type as well. For instance, a consumer group can be copied by finding a coordinator
(GroupCoordinatorRequest / FindCoordinatorRequest) and sending an OffsetCommitRequest. The
group will be auto-created with an empty protocol. Although this is arguably correct, the
group will now exist but not be  a proper 'consumer' group until later when there is a JoinGroupRequest.
Again, this can be confusing as far as categorization / visibility of the group is concerned.
A group can instead be copied more directly by creating a consumer and calling commitSync
(as kafka-consumer-groups.sh), but this does involve extra connections / requests and so is
a little slower when trying to keep a large number of groups in sync in real-time across clusters.
> If we want to make it easier to keep consumer groups consistent, options include allowing
groups to be explicitly created with a protocol type instead of only lazily, or for groups
created outside of JoinGroupRequest the coordinator can gain a broker config to set a default
protocol type for groups. This would default to 'consumer'. This sort of setting might be
confusing for users though, since implicitly created groups is certainly not the norm.

This message was sent by Atlassian JIRA

View raw message