kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6534) Consumer.poll may not trigger rebalance in time when there is a task migration
Date Wed, 07 Feb 2018 02:10:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Guozhang Wang updated KAFKA-6534:
    Issue Type: Bug  (was: Improvement)

> Consumer.poll may not trigger rebalance in time when there is a task migration
> ------------------------------------------------------------------------------
>                 Key: KAFKA-6534
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6534
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>            Priority: Major
> When Streams detect a task migration event in one of its thread, today it will always
let its trigger to call {{consumer.poll}} hoping it could trigger the rebalance and hence
clean up the records buffered from the partitions that on longer owned. However, because the
rebalance is based onĀ heartbeat responses which has a window of race, the rebalance is not
always guaranteed to be triggered when task migration happens. As a result it could cause
the records buffered in consumer to not be cleaned up and later be processed by Streams, realizing
it no longer belongs to the thread, causing:
> {code:java}
> java.lang.IllegalStateException: Record's partition does not belong to this partition-group.
> {code}
> Note this issue is only relevant when EOS is turned on, and based the default heartbeat.interval.ms
value (3 sec), the race likelihood should not be high.

This message was sent by Atlassian JIRA

View raw message