kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tamas Cserveny (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-2359) New consumer - partitions auto assigned only on poll
Date Thu, 29 Sep 2016 21:35:20 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534119#comment-15534119

Tamas Cserveny commented on KAFKA-2359:


We were struggling with this problem as well. We would like to use an external offset manager,
but cannot assign the offsets without calling pull(). 
To make matters works, pull(0) sometimes blocks more than 0ms! 

Our use-case would be:
Set topics/partitions
Seek offsets
and then call pull() in a loop

Of course as others mentioned, the seek is not working as expected. Of course I could manually
set the partition, but this would disable the partition management.



> New consumer - partitions auto assigned only on poll
> ----------------------------------------------------
>                 Key: KAFKA-2359
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2359
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Stevo Slavic
>            Priority: Minor
> In the new consumer I encountered unexpected behavior. After constructing {{KafkaConsumer}}
instance with configured consumer rebalance callback handler, and subscribing to a topic with
"consumer.subscribe(topic)", retrieving subscriptions would return empty set and callback
handler would not get called (no partitions ever assigned or revoked), no matter how long
instance was up.
> Then I found by inspecting {{KafkaConsumer}} code that partition assignment will only
be triggered on first {{poll}}, since {{pollOnce}} has:
> {noformat}
> // ensure we have partitions assigned if we expect to
> if (subscriptions.partitionsAutoAssigned())
>     coordinator.ensurePartitionAssignment();
> {noformat}
> I'm proposing to fix this by including same {{ensurePartitionAssignment}} fragment in
{{KafkaConsumer.subscriptions}} accessor as well.

This message was sent by Atlassian JIRA

View raw message