Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5A346200BAF for ; Mon, 31 Oct 2016 13:29:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 58C6E160AF0; Mon, 31 Oct 2016 12:29:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 94069160B05 for ; Mon, 31 Oct 2016 13:29:00 +0100 (CET) Received: (qmail 76011 invoked by uid 500); 31 Oct 2016 12:28:59 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 75953 invoked by uid 99); 31 Oct 2016 12:28:59 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Oct 2016 12:28:59 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 3E9A72C14F9 for ; Mon, 31 Oct 2016 12:28:59 +0000 (UTC) Date: Mon, 31 Oct 2016 12:28:59 +0000 (UTC) From: "Yiquan Zhou (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KAFKA-4348) On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 31 Oct 2016 12:29:01 -0000 [ https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622015#comment-15622015 ] Yiquan Zhou commented on KAFKA-4348: ------------------------------------ Did you get any "polled 0 records" before polling all the records? Because sometimes the consumer did poll all the records without interruption like what you got, but only after several seconds polling 0 record. This happens often right after the start of the server. But if you run the test for 4-5 more times you might get the same result as I did. If I set "enable.auto.commit" to true, here is an output that I got: Mon Oct 31 11:25:39 CET 2016 polled 0 records // first poll Mon Oct 31 11:25:40 CET 2016 polled 0 records Mon Oct 31 11:25:41 CET 2016 polled 0 records Mon Oct 31 11:25:42 CET 2016 polled 0 records Mon Oct 31 11:25:43 CET 2016 polled 0 records Mon Oct 31 11:25:43 CET 2016 polled 27171 records Mon Oct 31 11:25:43 CET 2016 polled 26886 records Mon Oct 31 11:25:43 CET 2016 polled 26886 records Mon Oct 31 11:25:44 CET 2016 polled 0 records Mon Oct 31 11:25:45 CET 2016 polled 0 records Mon Oct 31 11:25:46 CET 2016 polled 0 records Mon Oct 31 11:25:47 CET 2016 polled 0 records Mon Oct 31 11:25:48 CET 2016 polled 0 records Mon Oct 31 11:25:48 CET 2016 polled 26690 records Mon Oct 31 11:25:49 CET 2016 polled 26214 records Mon Oct 31 11:25:49 CET 2016 polled 26214 records Mon Oct 31 11:25:49 CET 2016 polled 26214 records Mon Oct 31 11:25:49 CET 2016 polled 13725 records Mon Oct 31 11:25:50 CET 2016 polled 0 records Mon Oct 31 11:25:51 CET 2016 polled 0 records Mon Oct 31 11:25:52 CET 2016 polled 0 records > On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server > ------------------------------------------------------------------------------------- > > Key: KAFKA-4348 > URL: https://issues.apache.org/jira/browse/KAFKA-4348 > Project: Kafka > Issue Type: Bug > Components: consumer > Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.1 > Environment: Mac OS X EI Capitan, Java 1.8.0_111 > Reporter: Yiquan Zhou > Labels: consumer, mac, polling > > Steps to reproduce: > 1. start the zookeeper and kafka server using the default properties from the distribution: > $ bin/zookeeper-server-start.sh config/zookeeper.properties > $ bin/kafka-server-start.sh config/server.properties > 2. create a Kafka consumer using the Java API KafkaConsumer.poll(long timeout). It polls the records from the server every second (timeout set to 1000) and prints the number of records polled. The code can be found here: https://gist.github.com/yiquanzhou/a94569a2c4ec8992444c83f3c393f596 > 3. use bin/kafka-verifiable-producer.sh to generate some messages: > $ bin/kafka-verifiable-producer.sh --topic connect-test --max-messages 200000 --broker-list localhost:9092 > wait until all 200k messages are generated and sent to the server. > 4. Run the consumer Java code. In the output console of the consumer, we can see that the consumer starts to poll some records, then it polls 0 records for several seconds before polling some more. like this: > polled 27160 records > polled 0 records > polled 0 records > polled 0 records > polled 0 records > polled 0 records > polled 26886 records > polled 26886 records > polled 0 records > polled 0 records > polled 0 records > polled 0 records > polled 0 records > polled 26701 records > polled 26214 records > The bug slows down the consumption of messages a lot. And in our use case, the consumer wrongly assumes that all messages are read from the topic. > It is only reproducible on Mac OS X but neither on Linux nor Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)