kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vahid Hashemian (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-6717) TopicPartition Assined twice to a consumer group for 2 consumer instances
Date Wed, 04 Apr 2018 18:42:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425964#comment-16425964
] 

Vahid Hashemian edited comment on KAFKA-6717 at 4/4/18 6:41 PM:
----------------------------------------------------------------

[~Yuancheng] Would it be possible to share a bit more about the use case?

I see consumers are subscribing to a pattern. Does this pattern frequently result in a different
set of topics? What were the topics that matched the regex when the issue occurred?

Anything extra you can share to better understand the issue and reproduce it would be a great
help. Are you able to easily reproduce the issue?

Thanks!


was (Author: vahid):
[~Yuancheng] Would it be possible to share a bit more about the use case?

I see consumers are subscribing to a pattern. Is this pattern frequently result in a different
set of topics? What were the topics that matched the regex when the issue occurred?

Anything extra you can share to better understand the issue and reproduce it would be a great
help. Are you able to easily reproduce the issue?

Thanks!

> TopicPartition Assined twice to a consumer group for 2 consumer instances 
> --------------------------------------------------------------------------
>
>                 Key: KAFKA-6717
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6717
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.11.0.1
>            Reporter: Yuancheng PENG
>            Priority: Major
>
> I'm using \{{StickyAssignor}} for consuming more than 100 topics with certain pattern.
> There are 10 consumers with the same group id.
> I expected that topic-partition to be assigned to only one consumer instance. However
some topic partitions are assigned twice in 2 different difference instance, hence the consumer
group process duplicate messages.
> {code:java}
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, Collections.singletonList(StickyAssignor.class));
> KafkaConsumer<Record, Record> c = new KafkaConsumer<>(props);
> c.subscribe(Pattern.compile(TOPIC_PATTERN), new NoOpConsumerRebalanceListener());
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message