Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Jason Brelloch <jb.bc.flk@gmail.com>
Date: Wed, 3 May 2017 10:46:42 -0400
Message-ID: <CAKY1MWp4vqR8DGsW9hb3dbg+-eEvkHYDCrv92iDWz+vtpedmOw@mail.gmail.com>
Subject: Tuning RocksDB
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=f403045c0e728a2c1d054e9fb96e
archived-at: Wed, 03 May 2017 14:46:48 -0000

--f403045c0e728a2c1d054e9fb96e
Content-Type: text/plain; charset=UTF-8

Hey all,

I am looking for some advice on tuning rocksDB for better performance in
Flink 1.2.  I created a pretty simple job with a single kafka source and
one flatmap function that just stores 50000 events in a single key of
managed keyed state and then drops everything else, to test checkpoint
performance.  Using a basic FsStateBackend configured as:

val backend = new FsStateBackend("file:///home/jason/flink/checkpoint")
env.setStateBackend(backend)

With about 30MB of state we see the checkpoints completing in 151ms.  Using
a RocksDBStateBackend configured as:

val backend = new RocksDBStateBackend("file:///home/jason/flink/checkpoint")
backend.setDbStoragePath("file:///home/jason/flink/rocksdb")
backend.setPredefinedOptions(PredefinedOptions.FLASH_SSD_OPTIMIZED)
env.setStateBackend(backend)

Running the same test the checkpoint takes 3 minutes 42 seconds.

I expect it to be slower, but that seems excessive.  I am also a little
confused as to when rocksDB and flink decide to write to disk, because
watching the database the .sst file wasn't created until significantly
after the checkpoint was completed, and the state had not changed.  Is
there anything I can do to increase the speed of the checkpoints, or
anywhere I can look to debug the issue?  (Nothing seems out of the ordinary
in the flink logs or rocksDB logs)

Thanks!

-- 
*Jason Brelloch* | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305
<http://www.bettercloud.com/>
Subscribe to the BetterCloud Monitor
<https://www.bettercloud.com/monitor?utm_source=bettercloud_email&utm_medium=email_signature&utm_campaign=monitor_launch>
-
Get IT delivered to your inbox

--f403045c0e728a2c1d054e9fb96e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey all,<div><br></div><div>I am looking for some advice o=
n tuning rocksDB for better performance in Flink 1.2.=C2=A0 I created a pre=
tty simple job with a single kafka source and one flatmap function that jus=
t stores 50000 events in a single key of managed keyed state and then drops=
 everything else, to test checkpoint performance.=C2=A0 Using a basic FsSta=
teBackend configured as:</div><div><br></div><div><div>val backend =3D new =
FsStateBackend(&quot;file:///home/jason/flink/checkpoint&quot;)</div><div>e=
nv.setStateBackend(backend)<br></div><div><br></div><div>With about 30MB of=
 state we see the checkpoints completing in 151ms.=C2=A0 Using a RocksDBSta=
teBackend configured as:</div><div><br></div><div>val backend =3D new Rocks=
DBStateBackend(&quot;file:///home/jason/flink/checkpoint&quot;)<br></div><d=
iv>backend.setDbStoragePath(&quot;file:///home/jason/flink/rocksdb&quot;)</=
div><div>backend.setPredefinedOptions(PredefinedOptions.FLASH_SSD_OPTIMIZED=
)</div><div>env.setStateBackend(backend)</div><div><br></div><div>Running t=
he same test the checkpoint takes 3 minutes 42 seconds.</div><div><br></div=
><div>I expect it to be slower, but that seems excessive.=C2=A0 I am also a=
 little confused as to when rocksDB and flink decide to write to disk, beca=
use watching the database the .sst file wasn&#39;t created until significan=
tly after the checkpoint was completed, and the state had not changed.=C2=
=A0 Is there anything I can do to increase the speed of the checkpoints, or=
 anywhere I can look to debug the issue? =C2=A0(Nothing seems out of the or=
dinary in the flink logs or rocksDB logs)</div><div><br></div><div>Thanks!<=
/div><div><br></div>-- <br><div class=3D"gmail_signature"><div dir=3D"ltr">=
<strong style=3D"color:rgb(32,53,68);font-family:&quot;open sans&quot;,helv=
etica,sans-serif;font-size:12px">Jason Brelloch</strong><span style=3D"colo=
r:rgb(32,53,68);font-family:&quot;open sans&quot;,helvetica,sans-serif;font=
-size:12px">=C2=A0| Product Developer</span><span style=3D"color:rgb(32,53,=
68);font-family:&quot;open sans&quot;,helvetica,sans-serif;font-size:12px">=
</span><div style=3D"color:rgb(32,53,68);font-family:&quot;open sans&quot;,=
helvetica,sans-serif;font-size:12px"><div style=3D"padding:5px 0px">3405 Pi=
edmont Rd. NE, Suite 325, Atlanta, GA 30305=C2=A0</div><a href=3D"http://ww=
w.bettercloud.com/" target=3D"_blank"><img alt=3D"" src=3D"https://www.bett=
ercloud.com/wp-content/uploads/email-sig.png" style=3D"width: 144px; margin=
: 10px 0px;"></a></div><div style=3D"color:rgb(32,53,68);font-family:&quot;=
open sans&quot;,helvetica,sans-serif;font-size:12px"><a href=3D"https://www=
.bettercloud.com/monitor?utm_source=3Dbettercloud_email&amp;utm_medium=3Dem=
ail_signature&amp;utm_campaign=3Dmonitor_launch" style=3D"color:rgb(0,171,2=
28);text-decoration:none" target=3D"_blank">Subscribe to the BetterCloud Mo=
nitor</a>=C2=A0- Get IT delivered to your inbox</div></div></div>
</div></div>

--f403045c0e728a2c1d054e9fb96e--