Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
References: 
 <CABtLTB=XvcDwkj0wD2PTSptZqS=GmTy7Bf9WKgh2=Cn7e6PeqA@mail.gmail.com>
In-Reply-To: 
 <CABtLTB=XvcDwkj0wD2PTSptZqS=GmTy7Bf9WKgh2=Cn7e6PeqA@mail.gmail.com>
From: Aljoscha Krettek <aljoscha@apache.org>
Date: Thu, 07 Apr 2016 08:48:19 +0000
Message-ID: 
 <CANMXwW1mwcywseKvdwxakBMRD6yUqLk2Mp+o1upatDQdyQsQjQ@mail.gmail.com>
Subject: Re: RocksDB state checkpointing is expensive?
To: "user@flink.apache.org" <user@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a114026d674bd5d052fe12428

--001a114026d674bd5d052fe12428
Content-Type: text/plain; charset=UTF-8

Hi,
you are right. Currently there is no incremental checkpointing and
therefore, at each checkpoint, we essentially copy the whole RocksDB
database to HDFS (or whatever filesystem you chose as a backup location).
As far as I know, Stephan will start working on adding support for
incremental snapshots this week or next week.

Cheers,
Aljoscha

On Thu, 7 Apr 2016 at 09:55 Krzysztof Zarzycki <k.zarzycki@gmail.com> wrote:

> Hi,
> I saw the documentation and source code of the state management with
> RocksDB and before I use it, I'm concerned of one thing: Am I right that
> currently when state is being checkpointed, the whole RocksDB state is
> snapshotted? There is no incremental, diff snapshotting, is it? If so, this
> seems to be unfeasible for keeping state counted in tens or hundreds of GBs
> (and you reach that size of a state, when you want to keep an embedded
> state of the streaming application instead of going out to Cassandra/Hbase
> or other DB). It will just cost too much to do snapshots of such large
> state.
>
> Samza as a good example to compare, writes every state change to Kafka
> topic, considering it a snapshot in the shape of changelog. Of course in
> the moment of app restart, recovering the state from the changelog would be
> too costly, that is why the changelog topic is compacted. Plus, I think
> Samza does a state snapshot from time to time anyway (but I'm not sure of
> that).
>
> Thanks for answering my doubts,
> Krzysztof
>
>

--001a114026d674bd5d052fe12428
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div>you are right. Currently there is no incremental c=
heckpointing and therefore, at each checkpoint, we essentially copy the who=
le RocksDB database to HDFS (or whatever filesystem you chose as a backup l=
ocation). As far as I know, Stephan will start working on adding support fo=
r incremental snapshots this week or next week.</div><div><br></div><div>Ch=
eers,</div><div>Aljoscha</div></div><br><div class=3D"gmail_quote"><div dir=
=3D"ltr">On Thu, 7 Apr 2016 at 09:55 Krzysztof Zarzycki &lt;<a href=3D"mail=
to:k.zarzycki@gmail.com">k.zarzycki@gmail.com</a>&gt; wrote:<br></div><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex"><div dir=3D"ltr">Hi,=C2=A0<div>I saw the document=
ation and source code of the state management with RocksDB and before I use=
 it, I&#39;m concerned of one thing: Am I right that currently when state i=
s being checkpointed, the whole RocksDB state is snapshotted? There is no i=
ncremental, diff snapshotting, is it? If so, this seems to be unfeasible fo=
r keeping state counted in tens or hundreds of GBs (and you reach that size=
 of a state, when you want to keep an embedded state of the streaming appli=
cation instead of going out to Cassandra/Hbase or other DB). It will just c=
ost too much to do snapshots of such large state.</div><div><br></div><div>=
Samza as a good example to compare, writes every state change to Kafka topi=
c, considering it a snapshot in the shape of changelog. Of course in the mo=
ment of app restart, recovering the state from the changelog would be too c=
ostly, that is why the changelog topic is compacted. Plus, I think Samza do=
es a state snapshot from time to time anyway (but I&#39;m not sure of that)=
.</div><div><br></div><div>Thanks for answering my doubts,=C2=A0</div></div=
><div dir=3D"ltr"><div>Krzysztof</div><div><br></div></div></blockquote></d=
iv>

--001a114026d674bd5d052fe12428--