kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "huxi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4348) On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server
Date Mon, 31 Oct 2016 09:41:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15621734#comment-15621734
] 

huxi commented on KAFKA-4348:
-----------------------------

interesting... I cannot reproduce on my Mac box. Here is what I got from you consumer code:
Polled 27171 records
Polled 26886 records
Polled 26886 records
Polled 26690 records
Polled 26214 records
Polled 26214 records
Polled 26214 records
Polled 13725 records
Polled 0 records
Polled 0 records
Polled 0 records
...

Could you please paste what you've got for the output once you set enable.auto.commit  to
true?

> On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server
> -------------------------------------------------------------------------------------
>
>                 Key: KAFKA-4348
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4348
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.1
>         Environment: Mac OS X EI Capitan, Java 1.8.0_111
>            Reporter: Yiquan Zhou
>              Labels: consumer, mac, polling
>
> Steps to reproduce:
> 1. start the zookeeper and kafka server using the default properties from the distribution:

> $ bin/zookeeper-server-start.sh config/zookeeper.properties
> $ bin/kafka-server-start.sh config/server.properties 
> 2. create a Kafka consumer using the Java API KafkaConsumer.poll(long timeout). It polls
the records from the server every second (timeout set to 1000) and prints the number of records
polled. The code can be found here: https://gist.github.com/yiquanzhou/a94569a2c4ec8992444c83f3c393f596
> 3. use bin/kafka-verifiable-producer.sh to generate some messages: 
> $ bin/kafka-verifiable-producer.sh --topic connect-test --max-messages 200000 --broker-list
localhost:9092
> wait until all 200k messages are generated and sent to the server. 
> 4. Run the consumer Java code. In the output console of the consumer, we can see that
the consumer starts to poll some records, then it polls 0 records for several seconds before
polling some more. like this:
> polled 27160 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26886 records
> polled 26886 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26701 records
> polled 26214 records
> The bug slows down the consumption of messages a lot. And in our use case, the consumer
wrongly assumes that all messages are read from the topic.
> It is only reproducible on Mac OS X but neither on Linux nor Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message