Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BD20B200C25 for ; Fri, 24 Feb 2017 17:44:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BBC25160B69; Fri, 24 Feb 2017 16:44:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B76CD160B62 for ; Fri, 24 Feb 2017 17:44:03 +0100 (CET) Received: (qmail 86784 invoked by uid 500); 24 Feb 2017 16:44:02 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 86775 invoked by uid 99); 24 Feb 2017 16:44:02 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Feb 2017 16:44:02 +0000 Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 3D4A41A00A2 for ; Fri, 24 Feb 2017 16:44:02 +0000 (UTC) Received: by mail-it0-f50.google.com with SMTP id 203so27958028ith.0 for ; Fri, 24 Feb 2017 08:44:02 -0800 (PST) X-Gm-Message-State: AMke39l4+APQhH/qIkjP/ZqpMfySDQFENOHRVO88qeZJ7CJsR+D4YmpbkJIJCrx4zWNP0Efnl0ebv3YkHK7+mw== X-Received: by 10.36.50.206 with SMTP id j197mr3529608ita.118.1487954637544; Fri, 24 Feb 2017 08:43:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.8.198 with HTTP; Fri, 24 Feb 2017 08:43:56 -0800 (PST) In-Reply-To: References: <1487753727469-11799.post@n4.nabble.com> <1487859615288-11831.post@n4.nabble.com> <1487941734848-11879.post@n4.nabble.com> <1487944701219-11882.post@n4.nabble.com> From: Stephan Ewen Date: Fri, 24 Feb 2017 17:43:56 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Checkpointing with RocksDB as statebackend To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a114aa33a9d35650549496f3f archived-at: Fri, 24 Feb 2017 16:44:04 -0000 --001a114aa33a9d35650549496f3f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Flink's state backends currently do a good number of "make sure this exists" operations on the file systems. Through Hadoop's S3 filesystem, that translates to S3 bucket list operations, where there is a limit in how many operation may happen per time interval. After that, S3 blocks. It seems that operations that are totally cheap on HDFS are hellishly expensive (and limited) on S3. It may be that you are affected by that. We are gradually trying to improve the behavior there and be more S3 aware. Both 1.3-SNAPSHOT and 1.2-SNAPSHOT already contain improvements there. Best, Stephan On Fri, Feb 24, 2017 at 4:42 PM, vinay patil wrote: > Hi Stephan, > > So do you mean that S3 is causing the stall , as I have mentioned in my > previous mail, I could not see any progress for 16minutes as checkpoints > were getting failed continuously. > > On Feb 24, 2017 8:30 PM, "Stephan Ewen [via Apache Flink User Mailing Lis= t > archive.]" <[hidden email] > > wrote: > >> Hi Vinay! >> >> True, the operator state (like Kafka) is currently not asynchronously >> checkpointed. >> >> While it is rather small state, we have seen before that on S3 it can >> cause trouble, because S3 frequently stalls uploads of even data amounts= as >> low as kilobytes due to its throttling policies. >> >> That would be a super important fix to add! >> >> Best, >> Stephan >> >> >> On Fri, Feb 24, 2017 at 2:58 PM, vinay patil <[hidden email] >> > wrote: >> >>> Hi, >>> >>> I have attached a snapshot for reference: >>> As you can see all the 3 checkpointins failed , for checkpoint ID 2 and >>> 3 it >>> is stuck at the Kafka source after 50% >>> (The data sent till now by Kafka source 1 is 65GB and sent by source 2 = is >>> 15GB ) >>> >>> Within 10minutes 15M records were processed, and for the next 16minutes >>> the >>> pipeline is stuck , I don't see any progress beyond 15M because of >>> checkpoints getting failed consistently. >>> >>> >> bble.com/file/n11882/Checkpointing_Failed.png> >>> >>> >>> >>> -- >>> View this message in context: http://apache-flink-user-maili >>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with- >>> RocksDB-as-statebackend-tp11752p11882.html >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >>> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend- >> tp11752p11885.html >> To start a new topic under Apache Flink User Mailing List archive., emai= l [hidden >> email] >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML >> >> > > ------------------------------ > View this message in context: Re: Checkpointing with RocksDB as > statebackend > > Sent from the Apache Flink User Mailing List archive. mailing list archiv= e > at > Nabble.com. > --001a114aa33a9d35650549496f3f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Flink's state backends currently do a good number of &= quot;make sure this exists" operations on the file systems. Through Ha= doop's S3 filesystem, that translates to S3 bucket list operations, whe= re there is a limit in how many operation may happen per time interval. Aft= er that, S3 blocks.

It seems that operations that are to= tally cheap on HDFS are hellishly expensive (and limited) on S3. It may be = that you are affected by that.

We are gradually tr= ying to improve the behavior there and be more S3 aware.

Both 1.3-SNAPSHOT and 1.2-SNAPSHOT already contain improvements ther= e.

Best,
Stephan


On Fri, Feb 24, 2017 = at 4:42 PM, vinay patil <vinay18.patil@gmail.com> wrot= e:

Hi Stephan,

So do you mean that S3 is causing the stall , as I have ment= ioned in my previous mail, I could not see any progress for 16minutes as ch= eckpoints were getting failed continuously.


= On Feb 24, 2017 8:30 PM, "Stephan Ewen [via Apache Flink User Mailing = List archive.]" <[hidden email]> wrote:
Hi Vinay!

True, the o= perator state (like Kafka) is currently not asynchronously checkpointed.

While it is rather small state, we have seen before = that on S3 it can cause trouble, because S3 frequently stalls uploads of ev= en data amounts as low as kilobytes due to its throttling policies.

That would be a super important fix to add!
Best,
Stephan


On Fri, Feb= 24, 2017 at 2:58 PM, vinay patil <[hidden email]> wrote:
Hi,

I have attached a snapshot for reference:
As you can see all the 3 checkpointins failed , for checkpoint ID 2 and 3 i= t
is stuck at the Kafka source after 50%
(The data sent till now by Kafka source 1 is 65GB and sent by source 2 is 15GB )

Within 10minutes 15M records were processed, and for the next 16minutes the=
pipeline is stuck , I don't see any progress beyond 15M because of
checkpoints getting failed consistently.

<http://apache-flink-user-mailing-list-archive.= 2336050.n4.nabble.com/file/n11882/Checkpointing_Failed.png>= ;



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabbl= e.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11882= .html
Sent from the Apache Flink= User Mailing List archive. mailing list archive at Nabble.com.

=09 =09 =09

To start a new topic under Apache Flink User Mailing List archive., email= [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML
=09 =09 =09
View this message in context: Re: Checkpointing with RocksDB a= s statebackend
Sent from the Apache Flink User Mailing List archiv= e. mailing list archive at Nabble.com.

--001a114aa33a9d35650549496f3f--