kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3664) When subscription set changes on new consumer, the partitions may be removed without offset being committed.
Date Fri, 06 May 2016 18:02:12 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274470#comment-15274470

Jiangjie Qin commented on KAFKA-3664:

[~hachikuji] I am not sure. If we still return the partitions of the unsubscribed topic when
user call {{assignment()}}, do we also return the records of those partitions? It seems weird
either way. 

The current behavior gives a good user experience to let the unsubscribe take effect immediately.
What surprising is the offsets are not committed during the following rebalance. Maybe we
can just fix that. I am thinking to add a new state in subscription state such as {{PendingUnsubscribe}}.
And the behavior after unsubscribe would be:
1. assignment() does not include the partitions of the unsubscribed topics.
2. subscription() does not include the unsubscribed topics.
3. poll() does not return messages from the unsubscribed topics.
4. In the rebalance listener, the partitions in {{PendingUnsubscribe}} state will be passed
in as revoked partitions.
5. When auto commit is enabled, during rebalance, we commit the offsets for partitions that
is in the assignment and the partitions that is pending unsubscribe.

Arguably after subscription changes, there is still an inconsistency between assignment()
and the partitions revoked passed to the rebalance listener, but it only affect the users
who have customized rebalance listener. Chances are those users do need to know the partitions
that are pending unsubscribe to checkpoint some states.

> When subscription set changes on new consumer, the partitions may be removed without
offset being committed.
> ------------------------------------------------------------------------------------------------------------
>                 Key: KAFKA-3664
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3664
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jiangjie Qin
>            Assignee: Vahid Hashemian
> When users are using group management, if they call consumer.subscribe() to change the
subscription, the removed subscriptions will be immediately removed and their offset will
not be commit. Also the revoked partitions passed to the ConsumerRebalanceListener.onPartitionsRevoked()
will not include those partitions. 

This message was sent by Atlassian JIRA

View raw message