kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lorenzo Alberton (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-761) Fetch request randomly returns 0 bytes when there's still data in the log
Date Mon, 18 Feb 2013 18:17:12 GMT
Lorenzo Alberton created KAFKA-761:
--------------------------------------

             Summary: Fetch request randomly returns 0 bytes when there's still data in the
log
                 Key: KAFKA-761
                 URL: https://issues.apache.org/jira/browse/KAFKA-761
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.7.2
         Environment: centos 6
            Reporter: Lorenzo Alberton
            Assignee: Neha Narkhede


I can't track the bug down in the code, but the behaviour has been hitting us for quite some
time (on Kafka 0.7.2): sometimes, a new Simple Consumer object, on a new Fetch Request, returns
an empty byte buffer (or a number of bytes considerably smaller than the buffer size), even
if there's more data on disk. 
This happens randomly, with no errors, at different offsets in the same topic. Different consumer
groups processing the same topic might get stuck at different offsets.
There seems to be no correlation between empty responses and the offset, or the buffer size,
or the log file size. At first we thought it only happened when reaching the end of a log
file, before jumping to the next file, but it doesn't seem to be the case. It often happens
in the middle of a file.

Simply retrying the Fetch Request multiple times works quite reliably, after 2 or 3 empty
fetches the consumer eventually gets unstuck (we check the size of the queue to see if there's
more data before retrying).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message