kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Babrou (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (KAFKA-6441) FetchRequest populates buffer of size MinBytes, even if response is smaller
Date Thu, 11 Jan 2018 05:16:02 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ivan Babrou updated KAFKA-6441:
    Comment: was deleted

(was: With consumer API Sarama is able to get multiple messages in one FetchResponse.

It doesn't seem right to get only one with API.)

> FetchRequest populates buffer of size MinBytes, even if response is smaller
> ---------------------------------------------------------------------------
>                 Key: KAFKA-6441
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6441
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Ivan Babrou
> We're using Sarama Go client as consumer, but I don't think it's relevant. Producer is
syslog-ng with Kafka output, I'm not quite sure which log format Kafka itself is using, but
I can assume, because that's what is set in topic settings.
> Our FetchRequest has minSize = 16MB, maxSize = 64, maxWait = 500ms. For a silly reason,
Kafka decides to reply with at least minSize buffer with just one 1KB log message. When Sarama
was using older consumer API, everything was okay. When we upgraded to consumer API,
consumer traffic for 125Mbit/s topic spiked to 55000Mbit/s on the wire and consumer wasn't
even able to keep up.
> 1KB message in a 16MB buffer is 1,600,000% overhead.
> I don't think there's any valid reason to do this.
> It's also mildly annoying that there is no tag in git, looking at changes is
harder than it should be.

This message was sent by Atlassian JIRA

View raw message