Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C60F8200C25 for ; Fri, 24 Feb 2017 19:52:30 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C4CD3160B69; Fri, 24 Feb 2017 18:52:30 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9E03C160B62 for ; Fri, 24 Feb 2017 19:52:29 +0100 (CET) Received: (qmail 94525 invoked by uid 500); 24 Feb 2017 18:52:28 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 94515 invoked by uid 99); 24 Feb 2017 18:52:28 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Feb 2017 18:52:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3F855C0781 for ; Fri, 24 Feb 2017 18:52:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.487 X-Spam-Level: ***** X-Spam-Status: No, score=5.487 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, HTML_MESSAGE=2, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id J_jwg3uUrQEk for ; Fri, 24 Feb 2017 18:52:26 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id EF8D45FAF1 for ; Fri, 24 Feb 2017 18:52:25 +0000 (UTC) Received: from mjoe.nabble.com (unknown [162.253.133.57]) by mwork.nabble.com (Postfix) with ESMTP id 9FB4F2EF99D5D for ; Fri, 24 Feb 2017 11:52:22 -0700 (MST) Date: Fri, 24 Feb 2017 10:47:28 -0800 (PST) From: vinay patil To: user@flink.apache.org Message-ID: In-Reply-To: References: <1487753727469-11799.post@n4.nabble.com> <1487859615288-11831.post@n4.nabble.com> <1487941734848-11879.post@n4.nabble.com> <1487944701219-11882.post@n4.nabble.com> Subject: Re: Checkpointing with RocksDB as statebackend MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_137934_899625068.1487962048469" archived-at: Fri, 24 Feb 2017 18:52:31 -0000 ------=_Part_137934_899625068.1487962048469 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi Stephan, To verify if S3 is making teh pipeline stall, I have replaced the S3 sink with HDFS and kept minimum pause between checkpoints to 5minutes, still I see the same issue with checkpoints getting failed. If I keep the pause time to 20 seconds, all checkpoints are completed , however there is a hit in overall throughput. Regards, Vinay Patil On Fri, Feb 24, 2017 at 10:09 PM, Stephan Ewen [via Apache Flink User Mailing List archive.] wrote: > Flink's state backends currently do a good number of "make sure this > exists" operations on the file systems. Through Hadoop's S3 filesystem, > that translates to S3 bucket list operations, where there is a limit in how > many operation may happen per time interval. After that, S3 blocks. > > It seems that operations that are totally cheap on HDFS are hellishly > expensive (and limited) on S3. It may be that you are affected by that. > > We are gradually trying to improve the behavior there and be more S3 aware. > > Both 1.3-SNAPSHOT and 1.2-SNAPSHOT already contain improvements there. > > Best, > Stephan > > > On Fri, Feb 24, 2017 at 4:42 PM, vinay patil <[hidden email] > > wrote: > >> Hi Stephan, >> >> So do you mean that S3 is causing the stall , as I have mentioned in my >> previous mail, I could not see any progress for 16minutes as checkpoints >> were getting failed continuously. >> >> On Feb 24, 2017 8:30 PM, "Stephan Ewen [via Apache Flink User Mailing >> List archive.]" <[hidden email] >> > wrote: >> >>> Hi Vinay! >>> >>> True, the operator state (like Kafka) is currently not asynchronously >>> checkpointed. >>> >>> While it is rather small state, we have seen before that on S3 it can >>> cause trouble, because S3 frequently stalls uploads of even data amounts as >>> low as kilobytes due to its throttling policies. >>> >>> That would be a super important fix to add! >>> >>> Best, >>> Stephan >>> >>> >>> On Fri, Feb 24, 2017 at 2:58 PM, vinay patil <[hidden email] >>> > wrote: >>> >>>> Hi, >>>> >>>> I have attached a snapshot for reference: >>>> As you can see all the 3 checkpointins failed , for checkpoint ID 2 and >>>> 3 it >>>> is stuck at the Kafka source after 50% >>>> (The data sent till now by Kafka source 1 is 65GB and sent by source 2 >>>> is >>>> 15GB ) >>>> >>>> Within 10minutes 15M records were processed, and for the next 16minutes >>>> the >>>> pipeline is stuck , I don't see any progress beyond 15M because of >>>> checkpoints getting failed consistently. >>>> >>>> >>> bble.com/file/n11882/Checkpointing_Failed.png> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-flink-user-maili >>>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with- >>>> RocksDB-as-statebackend-tp11752p11882.html >>>> Sent from the Apache Flink User Mailing List archive. mailing list >>>> archive at Nabble.com. >>>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11885.html >>> To start a new topic under Apache Flink User Mailing List archive., >>> email [hidden email] >>> >>> To unsubscribe from Apache Flink User Mailing List archive., click here. >>> NAML >>> >>> >> >> ------------------------------ >> View this message in context: Re: Checkpointing with RocksDB as >> statebackend >> >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive >> >> at Nabble.com. >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re- > Checkpointing-with-RocksDB-as-statebackend-tp11752p11891.html > To start a new topic under Apache Flink User Mailing List archive., email > ml-node+s2336050n1h83@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > > . > NAML > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11901.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. ------=_Part_137934_899625068.1487962048469 Content-Type: text/html; charset=UTF8 Content-Transfer-Encoding: quoted-printable
Hi Stephan,

To verify if S3 is= making teh pipeline stall, I have replaced the S3 sink with HDFS and kept = minimum pause between checkpoints to 5minutes, still I see the same issue w= ith checkpoints getting failed.

If I keep the=C2=A0 pause time= to 20 seconds, all checkpoints are completed , however there is a hit in o= verall throughput.



= Regards,
Vinay Patil
<= /div>

On Fri, Feb 24, 2017 at 10:09 PM, Stephan Ew= en [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
=
=09
Flink's state backends currently do a good number o= f "make sure this exists" operations on the file systems. Through= Hadoop's S3 filesystem, that translates to S3 bucket list operations, = where there is a limit in how many operation may happen per time interval. = After that, S3 blocks.

It seems that operations that are= totally cheap on HDFS are hellishly expensive (and limited) on S3. It may = be that you are affected by that.

We are gradually= trying to improve the behavior there and be more S3 aware.

<= /div>
Both 1.3-SNAPSHOT and 1.2-SNAPSHOT already contain improvements t= here.

Best,
Stephan

<= div class=3D"gmail_extra">
O= n Fri, Feb 24, 2017 at 4:42 PM, vinay patil <[hidden email]> wrote:

Hi Stephan,

So do you mean that S3 is causing the stall , as I have ment= ioned in my previous mail, I could not see any progress for 16minutes as ch= eckpoints were getting failed continuously.


On Feb 24, 2017 8:30 PM, "Stephan Ewen [via Apache Flink = User Mailing List archive.]" <[hidden email]> wrote:
=09
Hi Vinay!

True, the operator s= tate (like Kafka) is currently not asynchronously checkpointed.
<= br>
While it is rather small state, we have seen before that on S= 3 it can cause trouble, because S3 frequently stalls uploads of even data a= mounts as low as kilobytes due to its throttling policies.

That would be a super important fix to add!

Best,
Stephan


On Fri, Feb 24, 2017 at 2:58 PM,= vinay patil <[hidden email]> wrote:
Hi,

I have attached a snapshot for reference:
As you can see all the 3 checkpointins failed , for checkpoint ID 2 and 3 i= t
is stuck at the Kafka source after 50%
(The data sent till now by Kafka source 1 is 65GB and sent by source 2 is 15GB )

Within 10minutes 15M records were processed, and for the next 16minutes the=
pipeline is stuck , I don't see any progress beyond 15M because of
checkpoints getting failed consistently.

<http://apache-flink-user-mailing-list-archive.= 2336050.n4.nabble.com/file/n11882/Checkpointing_Failed.png>= ;



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabbl= e.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11882= .html
Sent from the Apache Flink User Mailing List archive. maili= ng list archive at Nabble.com.

=09 =09 =09 =09
=09
=09
=09
=09=09
If you reply to this email, your mess= age will be added to the discussion below:
=09=09http://apache-= flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpoi= nting-with-RocksDB-as-statebackend-tp11752p11885.html =09
 =09
=09=09To start a new topic under Apache Flink User Mailing List archive., e= mail [hidden email]=
=09=09To unsubscribe from Apache Flink User Mailing List archive., click here.
=09=09NAML<= /a> =09
=09 =09 =09
View this message in context: Re: Checkpointing with RocksDB as statebackend
Sent from the Ap= ache Flink User Mailing List archive. mailing list archive at Nabble.co= m.

=09 =09 =09 =09
=09
=09
=09
=09=09
If you reply to this email, your mess= age will be added to the discussion below:
=09=09http://apache-= flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Che= ckpointing-with-RocksDB-as-statebackend-tp11752p11891.html =09
=09
=09=09To start a new topic under Apache Flink User Mailing List archive., e= mail [hidden email]
=09=09To unsubscribe from Apache Flink User Mailing List archive., click here.<= br> =09=09NAML<= /a> =09

=09 =09 =09
View this message in context: Re: Checkpointing with RocksDB as statebackend=
Sent from the Apache Flink User Mailing List archive. mailing list ar= chive at Nabble.com.
------=_Part_137934_899625068.1487962048469--