From jira-return-8777-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Sat Jan 13 02:24:05 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 13EE3180621 for ; Sat, 13 Jan 2018 02:24:05 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 03FFF160C45; Sat, 13 Jan 2018 01:24:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 52C1E160C20 for ; Sat, 13 Jan 2018 02:24:04 +0100 (CET) Received: (qmail 40400 invoked by uid 500); 13 Jan 2018 01:24:03 -0000 Mailing-List: contact jira-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@kafka.apache.org Delivered-To: mailing list jira@kafka.apache.org Received: (qmail 40389 invoked by uid 99); 13 Jan 2018 01:24:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Jan 2018 01:24:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 115551A0846 for ; Sat, 13 Jan 2018 01:24:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.911 X-Spam-Level: X-Spam-Status: No, score=-99.911 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 7iBf08xxJqfk for ; Sat, 13 Jan 2018 01:24:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 995EC5F24C for ; Sat, 13 Jan 2018 01:24:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D3399E039B for ; Sat, 13 Jan 2018 01:24:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9797621E7F for ; Sat, 13 Jan 2018 01:24:00 +0000 (UTC) Date: Sat, 13 Jan 2018 01:24:00 +0000 (UTC) From: "Ivan Babrou (JIRA)" To: jira@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KAFKA-6441) FetchRequest populates buffer of size MinBytes, even if response is smaller MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324857#comment-16324857 ] Ivan Babrou commented on KAFKA-6441: ------------------------------------ Looks like the issue is in Sarama, which only reads one record batch: * https://github.com/Shopify/sarama/issues/1022 > FetchRequest populates buffer of size MinBytes, even if response is smaller > --------------------------------------------------------------------------- > > Key: KAFKA-6441 > URL: https://issues.apache.org/jira/browse/KAFKA-6441 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.11.0.1 > Reporter: Ivan Babrou > > We're using Sarama Go client as consumer, but I don't think it's relevant. Producer is syslog-ng with Kafka output, I'm not quite sure which log format Kafka itself is using, but I can assume 0.11.0.0, because that's what is set in topic settings. > Our FetchRequest has minSize = 16MB, maxSize = 64, maxWait = 500ms. For a silly reason, Kafka decides to reply with at least minSize buffer with just one 1KB log message. When Sarama was using older consumer API, everything was okay. When we upgraded to 0.11.0.0 consumer API, consumer traffic for 125Mbit/s topic spiked to 55000Mbit/s on the wire and consumer wasn't even able to keep up. > 1KB message in a 16MB buffer is 1,600,000% overhead. > I don't think there's any valid reason to do this. > It's also mildly annoying that there is no tag 0.11.0.1 in git, looking at changes is harder than it should be. -- This message was sent by Atlassian JIRA (v6.4.14#64029)