Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Josh <jofo90@gmail.com>
Date: Mon, 24 Oct 2016 18:06:22 +0100
Message-ID: <CABmO9D9N-+H8U_mEcPAqgCQenHsHo6aPQCou==FQK+DD=TMO+g@mail.gmail.com>
Subject: Checkpointing large RocksDB state to S3 - tips?
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0d451a511f37053f9f6905
archived-at: Mon, 24 Oct 2016 17:06:33 -0000

--94eb2c0d451a511f37053f9f6905
Content-Type: text/plain; charset=UTF-8

Hi all,

I'm running Flink on EMR/YARN with 2x m3.xlarge instances and am
checkpointing a fairly large RocksDB state to S3.

I've found that when the state size hits 10GB, the checkpoint takes around
6 minutes, according to the Flink dashboard. Originally my checkpoint
interval was 5 minutes for the job, but I've found that the YARN container
crashes (I guess because the checkpoint time is greater than the checkpoint
interval), so have now decreased the checkpoint frequency to every 10
minutes.

I was just wondering if anyone has any tips about how to reduce the
checkpoint time. Taking 6 minutes to checkpoint ~10GB state means it's
uploading at ~30MB/sec. I believe the m3.xlarge instances should have
around 125MB/sec network bandwidth each, so I think the bottleneck is S3.
Since there are 2 instances, I'm not sure if that means each instance is
uploading at 15MB/sec - do the state uploads get shared equally among the
instances, assuming the state is split equally between the task managers?

If the state upload is split between the instances, perhaps the only way to
speed up the checkpoints is to add more instances and task managers, and
split the state equally among the task managers?

Also just wondering - is there any chance the incremental checkpoints work
will be complete any time soon?

Thanks,
Josh

--94eb2c0d451a511f37053f9f6905
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;m running Flink on EMR/YA=
RN with 2x m3.xlarge instances and am checkpointing a fairly large RocksDB =
state to S3.</div><div><br></div><div>I&#39;ve found that when the state si=
ze hits 10GB, the checkpoint takes around 6 minutes, according to the Flink=
 dashboard. Originally my checkpoint interval was 5 minutes for the job, bu=
t I&#39;ve found that the YARN container crashes (I guess because the check=
point time is greater than the checkpoint interval), so have now decreased =
the checkpoint frequency to every 10 minutes.</div><div><br></div><div>I wa=
s just wondering if anyone has any tips about how to reduce the checkpoint =
time. Taking 6 minutes to checkpoint ~10GB state means it&#39;s uploading a=
t ~30MB/sec. I believe the m3.xlarge instances should have around 125MB/sec=
 network bandwidth each, so I think the bottleneck is S3.=C2=A0</div><div>S=
ince there are 2 instances, I&#39;m not sure if that means each instance is=
 uploading at 15MB/sec - do the state uploads get shared equally among the =
instances, assuming the state is split equally between the task managers?<b=
r></div><div><br></div><div>If the state upload is split between the instan=
ces, perhaps the only way to speed up the checkpoints is to add more instan=
ces and task managers, and split the state equally among the task managers?=
<br></div><div><br></div><div>Also just wondering - is there any chance the=
 incremental checkpoints work will be complete any time soon?<br></div><div=
><br></div><div>Thanks,</div><div>Josh</div></div>

--94eb2c0d451a511f37053f9f6905--