kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6534) Consumer.poll may not trigger rebalance in time when there is a task migration
Date Mon, 05 Feb 2018 19:52:00 GMT
Guozhang Wang created KAFKA-6534:
------------------------------------

             Summary: Consumer.poll may not trigger rebalance in time when there is a task
migration
                 Key: KAFKA-6534
                 URL: https://issues.apache.org/jira/browse/KAFKA-6534
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Guozhang Wang


When Streams detect a task migration event in one of its thread, today it will always let
its trigger to call {{consumer.poll}} hoping it could trigger the rebalance and hence clean
up the records buffered from the partitions that on longer owned. However, because the rebalance
is based on HB responses which has a window of race, the rebalance is not always guaranteed
to be triggered when task migration happens. As a result it could cause the records buffered
in consumer to not be cleaned up and later be processed by Streams, realizing it no longer
belongs to the thread, causing:


{code}
java.lang.IllegalStateException: Record's partition does not belong to this partition-group.
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message