kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6717) TopicPartition Assined twice to a consumer group for 2 consumer instances
Date Thu, 29 Mar 2018 04:59:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418435#comment-16418435

Ewen Cheslack-Postava commented on KAFKA-6717:

Are both assigned the partition in the same generation? As consumers join the group, it will
rebalance and change the assignment. If you are just looking at which consumers are assigned
which partitions, it could appear that two of them are assigned the same partitions at the
same time.

I see you are using the NoOpConsumerRebalanceListener. With this, you wouldn't see when partitions
were assigned or revoked. What are you doing to verify that both consumer instances are assigned
the same partitions at the same time?

Without correct handling of partition assignments and revocation, you definitely could see
data processed twice. In fact, without additional steps taken to ensure no duplicates, *at
least once* handling is what Kafka consumers would normally provide as long as they handle
offset commits properly.

> TopicPartition Assined twice to a consumer group for 2 consumer instances 
> --------------------------------------------------------------------------
>                 Key: KAFKA-6717
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6717
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Yuancheng PENG
>            Priority: Major
> I'm using \{{StickyAssignor}} for consuming more than 100 topics with certain pattern.
> There are 10 consumers with the same group id.
> I expected that topic-partition to be assigned to only one consumer instance. However
some topic partitions are assigned twice in 2 different difference instance, hence the consumer
group process duplicate messages.
> {code:java}
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, Collections.singletonList(StickyAssignor.class));
> KafkaConsumer<Record, Record> c = new KafkaConsumer<>(props);
> c.subscribe(Pattern.compile(TOPIC_PATTERN), new NoOpConsumerRebalanceListener());
> {code}

This message was sent by Atlassian JIRA

View raw message