Mailing-List: contact user-help@kudu.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@kudu.apache.org
MIME-Version: 1.0
In-Reply-To: <CADY20s5UZN226ysQW6oRv4NPJbyGq3gx7+ecUJt22s9CM=h2mA@mail.gmail.com>
References: <CAOHFFGSnDRYOw8+mrj6zNtNX+fr9J6eXMLx+_aM1gZ_9O9U79g@mail.gmail.com>
 <CAAnRTpJkDAEaopaG35WzfQdD=NufwA9OWDMNjcf2Ahvdyd-tDA@mail.gmail.com> <CADY20s5UZN226ysQW6oRv4NPJbyGq3gx7+ecUJt22s9CM=h2mA@mail.gmail.com>
From: David Alves <davidralves@gmail.com>
Date: Mon, 24 Apr 2017 11:06:02 -0700
Message-ID: <CAAnRTpLqCjami7aJMg9uNrdkDW-Tdtuk6EbTN_KZD_N4q7UMVA@mail.gmail.com>
Subject: Re: Bad insert performance of java kudu-client
To: user <user@kudu.apache.org>
Content-Type: multipart/alternative; boundary=001a113dd7ae064767054ded770e
archived-at: Mon, 24 Apr 2017 18:06:29 -0000

--001a113dd7ae064767054ded770e
Content-Type: text/plain; charset=UTF-8

I just became aware that AUTOFLUSH_BACKGROUND has been available for longer
on the java client, so likely not the cause after all. So never mind my
suggestion.

-david

On Mon, Apr 24, 2017 at 10:56 AM, Todd Lipcon <todd@cloudera.com> wrote:

> I vaguely recall some bug in earlier versions of the Java client where
> 'shutdown' wouldn't properly block on the data being flushed. So it's
> possible in 1.0.x and below, you're not actually measuring the full amount
> of time to write all the data, whereas when the bug is fixed, you are.
>
> I'll see if I can repro this locally as well using your code.
>
> -Todd
>
> On Mon, Apr 24, 2017 at 10:49 AM, David Alves <davidralves@gmail.com>
> wrote:
>
>> Hi Pavel
>>
>>   Interesting, Thanks for sharing those numbers.
>>   I assume you weren't using AUTOFLUSH_BACKGROUND for the first versions
>> you tested (don't think it was available then iirc).
>>   Could you try without in the last version and see how the numbers
>> compare?
>>   We'd be happy to help track down the reason for this perf regression.
>>
>> Best
>> David
>>
>> On Mon, Apr 24, 2017 at 4:58 AM, Pavel Martynov <mr.xkurt@gmail.com>
>> wrote:
>>
>>> Hi, I ran into the fact that I can not achieve high insertion speed and
>>> I start to experiment with https://github.com/cloude
>>> ra/kudu-examples/tree/master/java/insert-loadgen.
>>> My slightly modified code (recreation of table on startup + duration
>>> measuring): https://gist.github.com/xkrt/9405a2eeb98a56288b7
>>> c5a7d817097b4.
>>> On every run I change kudu-client version, results:
>>>
>>> kudu-client-ver  perf
>>> 0.10             Duration: 626 ms, 79872/sec
>>> 1.0.0            Duration: 622 ms, 80385 inserts/sec
>>> 1.0.1            Duration: 630 ms, 79365 inserts/sec
>>> 1.1.0            Duration: 11703 ms, 4272 inserts/sec
>>> 1.3.1            Duration: 12317 ms, 4059 inserts/sec
>>>
>>> As can you see there was a great degradation between 1.0.1 and 1.1.0
>>> (about a ~20 times!).
>>> What could be a problem, how can I fix it? (actually I interested in
>>> kudu-spark, so probably using of kudu-client 1.0.1 is not right solution?).
>>>
>>> My test cluster: 3 hosts with master and tserver on each (3 masters and
>>> 3 tservers overall).
>>> No extra settings, flags used:
>>> fs_wal_dir
>>> fs_data_dirs
>>> master_addresses
>>> tserver_master_addrs
>>>
>>>
>>> --
>>> with best regards, Pavel Martynov
>>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--001a113dd7ae064767054ded770e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I just became aware that AUTOFLUSH_BACKGROUND has been ava=
ilable for longer on the java client, so likely not the cause after all. So=
 never mind my suggestion.<div><br></div><div>-david</div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Apr 24, 2017 at 10:=
56 AM, Todd Lipcon <span dir=3D"ltr">&lt;<a href=3D"mailto:todd@cloudera.co=
m" target=3D"_blank">todd@cloudera.com</a>&gt;</span> wrote:<br><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><div dir=3D"ltr">I vaguely recall some bug in earlier v=
ersions of the Java client where &#39;shutdown&#39; wouldn&#39;t properly b=
lock on the data being flushed. So it&#39;s possible in 1.0.x and below, yo=
u&#39;re not actually measuring the full amount of time to write all the da=
ta, whereas when the bug is fixed, you are.<div><br></div><div>I&#39;ll see=
 if I can repro this locally as well using your code.</div><div><br></div><=
div>-Todd</div></div><div class=3D"gmail_extra"><div><div class=3D"h5"><br>=
<div class=3D"gmail_quote">On Mon, Apr 24, 2017 at 10:49 AM, David Alves <s=
pan dir=3D"ltr">&lt;<a href=3D"mailto:davidralves@gmail.com" target=3D"_bla=
nk">davidralves@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex"><div dir=3D"ltr">Hi Pavel<div><br></div><div>=C2=A0 Interesting, Tha=
nks for sharing those numbers.</div><div>=C2=A0 I assume you weren&#39;t us=
ing AUTOFLUSH_BACKGROUND for the first versions you tested (don&#39;t think=
 it was available then iirc).</div><div>=C2=A0 Could you try without in the=
 last version and see how the numbers compare?</div><div>=C2=A0 We&#39;d be=
 happy to help track down the reason for this perf regression.</div><div><b=
r></div><div>Best</div><span class=3D"m_8780743174601399182HOEnZb"><font co=
lor=3D"#888888"><div>David</div></font></span></div><div class=3D"m_8780743=
174601399182HOEnZb"><div class=3D"m_8780743174601399182h5"><div class=3D"gm=
ail_extra"><br><div class=3D"gmail_quote">On Mon, Apr 24, 2017 at 4:58 AM, =
Pavel Martynov <span dir=3D"ltr">&lt;<a href=3D"mailto:mr.xkurt@gmail.com" =
target=3D"_blank">mr.xkurt@gmail.com</a>&gt;</span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div dir=3D"ltr">Hi, I ran into the fact that I can not a=
chieve high insertion speed and I start=C2=A0to experiment with=C2=A0<a hre=
f=3D"https://github.com/cloudera/kudu-examples/tree/master/java/insert-load=
gen" target=3D"_blank">https://github.com/cloude<wbr>ra/kudu-examples/tree/=
master/j<wbr>ava/insert-loadgen</a>.<div>My slightly modified code (recreat=
ion of table on startup + duration measuring):=C2=A0<a href=3D"https://gist=
.github.com/xkrt/9405a2eeb98a56288b7c5a7d817097b4" target=3D"_blank">https:=
//gist.githu<wbr>b.com/xkrt/9405a2eeb98a56288b7<wbr>c5a7d817097b4</a>.</div=
><div>On every=C2=A0run I change kudu-client version, results:</div><div><b=
r></div><div><div>kudu-client-ver =C2=A0perf</div><div>0.10 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Duration: 626 ms, 79872/sec</div><div>1.0.0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Duration: 622 ms, 80385 inserts/sec</=
div><div>1.0.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Duration: 630 ms, 7=
9365 inserts/sec</div><div>1.1.0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0D=
uration: 11703 ms, 4272 inserts/sec</div><div>1.3.1 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0Duration: 12317 ms, 4059 inserts/sec</div></div><div><b=
r></div><div>As can you see there was a great degradation between 1.0.1 and=
 1.1.0 (about a ~20 times!).</div><div>What could be a problem, how can I f=
ix it? (actually I interested in kudu-spark, so probably using of kudu-clie=
nt 1.0.1 is not right solution?).</div><div><br></div><div>My test cluster:=
 3 hosts with master and tserver on each (3 masters and 3 tservers overall)=
.</div><div>No extra settings, flags used:</div><div>fs_wal_dir<br></div><d=
iv>fs_data_dirs<br></div><div>master_addresses<br></div><div>tserver_master=
_addrs</div><span class=3D"m_8780743174601399182m_1447058198777092552HOEnZb=
"><font color=3D"#888888"><div><br></div><div><br></div><div><div>-- <br><d=
iv class=3D"m_8780743174601399182m_1447058198777092552m_-562127496792506270=
2gmail-m_-1194545999651191921gmail_signature">with best regards, Pavel Mart=
ynov<br></div>
</div></div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div></div><=
/div><span class=3D"HOEnZb"><font color=3D"#888888">-- <br><div class=3D"m_=
8780743174601399182gmail_signature" data-smartmail=3D"gmail_signature">Todd=
 Lipcon<br>Software Engineer, Cloudera</div>
</font></span></div>
</blockquote></div><br></div>

--001a113dd7ae064767054ded770e--