Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <76A7B822-1DEB-4593-A639-C70FF887137A@inria.fr>
References: <76A7B822-1DEB-4593-A639-C70FF887137A@inria.fr>
From: Jay Kreps <jay@confluent.io>
Date: Thu, 20 Jul 2017 12:06:47 -0700
Message-ID: <CAOeJiJj8g_iYkbPPF3vmF9fa2Psn43VTS766c-+4ccgj2ET_Kg@mail.gmail.com>
Subject: Re: Consumer throughput drop
To: "users@kafka.apache.org" <users@kafka.apache.org>
Cc: "dev@kafka.apache.org" <dev@kafka.apache.org>,
	Ovidiu Cristian Marcu <ovidiu-cristian.marcu@inria.fr>
Content-Type: multipart/related; boundary="001a1144377e5447900554c4736a"
archived-at: Thu, 20 Jul 2017 19:07:00 -0000

--001a1144377e5447900554c4736a
Content-Type: multipart/alternative; boundary="001a1144377e54478d0554c47369"

--001a1144377e54478d0554c47369
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I suspect this is on Linux right?

The way Linux works is it uses a percent of memory to buffer new writes, at
a certain point it thinks it has too much buffered data and it gives high
priority to writing that out. The good news about this is that the writes
are very linear, well layed out, and high-throughput. The problem is that
it leads to a bit of see-saw behavior.

Now obviously the drop in performance isn't wrong. When your disk is
writing data out it is doing work and obviously the read throughput will be
higher when you are just reading and not writing then when you are doing
both reading and writing simultaneously. So obviously you can't get the
no-writing performance when you are also writing (unless you add I/O
capacity).

But still these big see-saws in performance are not ideal. You'd rather
have more constant performance all the time rather than have linux bounce
back and forth from writing nothing and then frantically writing full bore.
Fortunately linux provides a set of pagecache tuning parameters that let
you control this a bit.

I think these docs cover some of the parameters:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/ht=
ml/Performance_Tuning_Guide/s-memory-tunables.html

-Jay

On Thu, Jul 20, 2017 at 10:24 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.marcu@inria.fr> wrote:

> Hi guys,
>
> I=E2=80=99m relatively new to Kafka=E2=80=99s world. I have an issue I de=
scribe below,
> maybe you can help me understand this behaviour.
>
> I=E2=80=99m running a benchmark using the following setup: one producer s=
ends data
> to a topic and concurrently one consumer pulls and writes it to another
> topic.
> Measuring the consumer throughput, I observe values around 500K records/s
> only until the system=E2=80=99s cache gets filled - from this moment the =
consumer
> throughout drops to ~200K (2.5 times lower).
> Looking at disk usage, I observe disk read I/O which corresponds to the
> moment the consumer throughout drops.
> After some time, I kill the producer and immediately I observe the
> consumer throughput goes up to initial values ~ 500K records/s.
>
> What can I do to avoid this throughput drop?
>
> Attached an image showing disk I/O and CPU usage. I have about 128GB RAM
> on that server which gets filled at time ~2300.
>
> Thanks,
> Ovidiu
>
>

--001a1144377e54478d0554c47369
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I suspect this is on Linux right?<div><br></div><div>The w=
ay Linux works is it uses a percent of memory to buffer new writes, at a ce=
rtain point it thinks it has too much buffered data and it gives high prior=
ity to writing that out. The good news about this is that the writes are ve=
ry linear, well layed out, and high-throughput. The problem is that it lead=
s to a bit of see-saw behavior.</div><div><br></div><div>Now obviously the =
drop in performance isn&#39;t wrong. When your disk is writing data out it =
is doing work and obviously the read throughput will be higher when you are=
 just reading and not writing then when you are doing both reading and writ=
ing simultaneously. So obviously you can&#39;t get the no-writing performan=
ce when you are also writing (unless you add I/O capacity).</div><div><br><=
/div><div>But still these big see-saws in performance are not ideal. You=
9;d rather have more constant performance all the time rather than have lin=
ux bounce back and forth from writing nothing and then frantically writing =
full bore. Fortunately linux provides a set of pagecache tuning parameters =
that let you control this a bit.=C2=A0</div><div><br></div><div>I think the=
se docs cover some of the parameters:</div><div><a href=3D"https://access.r=
edhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_T=
uning_Guide/s-memory-tunables.html">https://access.redhat.com/documentation=
/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-tu=
nables.html</a><br></div><div><br></div><div>-Jay</div></div><div class=3D"=
gmail_extra"><br><div class=3D"gmail_quote">On Thu, Jul 20, 2017 at 10:24 A=
M, Ovidiu-Cristian MARCU <span dir=3D"ltr">&lt;<a href=3D"mailto:ovidiu-cri=
stian.marcu@inria.fr" target=3D"_blank">ovidiu-cristian.marcu@inria.fr</a>&=
gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap=
:break-word">Hi guys,<div><br></div><div>I=E2=80=99m relatively new to Kafk=
a=E2=80=99s world. I have an issue I describe below, maybe you can help me =
understand this behaviour.</div><div><br></div><div>I=E2=80=99m running a b=
enchmark using the following setup: one producer sends data to a topic and =
concurrently one consumer pulls and writes it to another topic.</div><div>M=
easuring the consumer throughput, I observe values around 500K records/s on=
ly until the system=E2=80=99s cache gets filled - from this moment the cons=
umer throughout drops to ~200K (2.5 times lower).</div><div>Looking at disk=
 usage, I observe disk read I/O which corresponds to the moment the consume=
r throughout drops.</div><div>After some time, I kill the producer and imme=
diately I observe the consumer throughput goes up to initial values ~ 500K =
records/s.</div><div><br></div><div>What can I do to avoid this throughput =
drop?</div><div><br></div><div>Attached an image showing disk I/O and CPU u=
sage. I have about 128GB RAM on that server which gets filled at time ~2300=
.</div><div><br></div><div>Thanks,</div><div>Ovidiu</div><div><br></div><di=
v><img id=3D"m_-3646119185184359713BBC96783-5CBE-46B6-A633-F060A4E5F84A" he=
ight=3D"689" width=3D"1109" src=3D"cid:17A3A338-06F2-4D68-A96B-21525B30564C=
@home"></div></div></blockquote></div><br></div>

--001a1144377e54478d0554c47369--

--001a1144377e5447900554c4736a--