Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C16C9200C40 for ; Thu, 9 Mar 2017 03:27:35 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C0108160B86; Thu, 9 Mar 2017 02:27:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 99EB3160B83 for ; Thu, 9 Mar 2017 03:27:34 +0100 (CET) Received: (qmail 71344 invoked by uid 500); 9 Mar 2017 02:27:33 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 71334 invoked by uid 99); 9 Mar 2017 02:27:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Mar 2017 02:27:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 3A5621A039F for ; Thu, 9 Mar 2017 02:27:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.487 X-Spam-Level: ***** X-Spam-Status: No, score=5.487 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, HTML_MESSAGE=2, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id M9AAjLEPCw16 for ; Thu, 9 Mar 2017 02:27:30 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 16F255F64B for ; Thu, 9 Mar 2017 02:27:30 +0000 (UTC) Received: from mjoe.nabble.com (unknown [162.253.133.57]) by mwork.nabble.com (Postfix) with ESMTP id 0A1C431AFD7B4 for ; Wed, 8 Mar 2017 19:27:29 -0700 (MST) Date: Wed, 8 Mar 2017 18:21:19 -0800 (PST) From: vinay patil To: user@flink.apache.org Message-ID: In-Reply-To: References: <2778A8A8-9432-4881-98EB-A17426CCC031@data-artisans.com> <8552427B-F100-411C-8B15-EBD677A58CE4@data-artisans.com> Subject: Re: Frequent Full GC's in case of FSStateBackend MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1848_1566169272.1489026079533" archived-at: Thu, 09 Mar 2017 02:27:35 -0000 ------=_Part_1848_1566169272.1489026079533 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi Sai, If you are sure that your state will not exceed the memory limit of nodes then you should consider FSStatebackend otherwise you should go for RocksDB What is the configuration of your cluster ? On Mar 9, 2017 7:31 AM, "saiprasad mishra [via Apache Flink User Mailing List archive.]" wrote: > Hi All > > I am also seeing issues with FsStateBackend as it stalls coz of full gc. > We have very large state, > Does this mean the below doc should not claim that FsStateBackend is > encouraged for large state. > > https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_ > backends.html#the-fsstatebackend > > Regards > Sai > > On Fri, Feb 10, 2017 at 6:19 AM, Stefan Richter <[hidden email] > > wrote: > >> Async snapshotting is the default. >> >> Am 10.02.2017 um 14:03 schrieb vinay patil <[hidden email] >> >: >> >> Hi Stephan, >> >> Thank you for the clarification. >> Yes with RocksDB I don't see Full GC happening, also I am using Flink >> 1.2.0 version and I have set the statebackend in flink-conf.yaml file to >> rocksdb, so by default does this do asynchronous checkpointing or I have to >> specify it at the job level ? >> >> Regards, >> Vinay Patil >> >> On Fri, Feb 10, 2017 at 4:16 PM, Stefan Richter [via Apache Flink User >> Mailing List archive.] <[hidden email]> wrote: >> >>> Hi, >>> >>> FSStateBackend operates completely on-heap and only snapshots for >>> checkpoints go against the file system. This is why the backend is >>> typically faster for small states, but can become problematic for larger >>> states. If your state exceeds a certain size, you should strongly consider >>> to use RocksDB as backend. In particular, RocksDB also offers asynchronous >>> snapshots which is very valuable to keep stream processing running for >>> large state. RocksDB works on native memory/disk, so there is no GC to >>> observe. For cases in which your state fits in memory but GC is a problem >>> you could try using the G1 garbage collector which offers better >>> performance for the FSStateBackend than the default. >>> >>> Best, >>> Stefan >>> >>> >>> Am 10.02.2017 um 11:16 schrieb Vinay Patil <[hidden email] >>> >: >>> >>> Hi, >>> >>> I am doing performance test for my pipeline keeping FSStateBackend, I >>> have observed frequent Full GC's after processing 20M records. >>> >>> When I did memory analysis using MAT, it showed that the many objects >>> maintained by Flink state are live. >>> >>> Flink keeps the state in memory even after checkpointing , when does >>> this state gets removed / GC. (I am using window operator in which the DTO >>> comes as input) >>> >>> Also why does Flink keep the state in memory after checkpointing ? >>> >>> P.S Using RocksDB is not causing Full GC at all. >>> >>> Regards, >>> Vinay Patil >>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/Frequent-Full-GC-s-in-case-of-FSStateBackend-tp11564p11565.html >>> To start a new topic under Apache Flink User Mailing List archive., >>> email [hidden email] >>> To unsubscribe from Apache Flink User Mailing List archive., click here. >>> NAML >>> >>> >> >> >> ------------------------------ >> View this message in context: Re: Frequent Full GC's in case of >> FSStateBackend >> >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive >> >> at Nabble.com. >> >> >> > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Frequent-Full-GC-s-in-case-of-FSStateBackend-tp11564p12126. > html > To start a new topic under Apache Flink User Mailing List archive., email > ml-node+s2336050n1h83@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > > . > NAML > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Frequent-Full-GC-s-in-case-of-FSStateBackend-tp11564p12127.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. ------=_Part_1848_1566169272.1489026079533 Content-Type: text/html; charset=UTF8 Content-Transfer-Encoding: quoted-printable

Hi Sai,

If you are sure that your state will not exceed the memory l= imit of nodes then you should consider FSStatebackend otherwise you should = go for RocksDB

What is the configuration of your cluster ?


On Mar 9, 2017 7:= 31 AM, "saiprasad mishra [via Apache Flink User Mailing List archive.]= " <[hidden email]> wr= ote:
=09
Hi All

I am also seeing issues with= =C2=A0FsStateBackend as it stalls coz of full gc. We have very large state,=
Does this mean the below doc should not claim that FsStateBacken= d is encouraged for large state.


Regards
Sai

On Fri, Feb 10, 2017 at 6:19 AM, Stefan Richter <[= hidden email]> wrote:
Async sn= apshotting is the default.=C2=A0

=
Am 10.02.2017 um 14:03 schrieb vinay patil <[hidden email]>:

Hi Stephan,

Thank you for the clarification.
Ye= s with RocksDB I don't see Full GC happening, also I am using Flink 1.2= .0 version and I have set the statebackend in flink-conf.yaml file to rocks= db, so by default does this do asynchronous checkpointing or I have to spec= ify it at the job level =C2=A0?

Regards,
Vinay Patil
=

On Fri, Feb 10, 2017 at 4:16 PM, Stefan Rich= ter [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
=09
Hi,

FSStateBackend operates completely on-= heap and only snapshots for checkpoints go against the file system. This is= why the backend is typically faster for small states, but can become probl= ematic for larger states. If your state exceeds a certain size, you should = strongly consider to use RocksDB as backend. In particular, RocksDB also of= fers asynchronous snapshots which is very valuable to keep stream processin= g running for large state. RocksDB works on native memory/disk, so there is= no GC to observe. For cases in which your state fits in memory but GC is a= problem you could try using the G1 garbage collector which offers better p= erformance for the FSStateBackend than the default.

Best,
Stefan


Am 10.02.2017 um 11:16 schrieb Vinay Pat= il <[hidden email= ]>:

Hi,
=
I am doing performance test for my pipeline keeping FSStateBackend, I h= ave observed frequent Full GC's after processing 20M records.

Wh= en I did memory analysis using MAT, it showed that the many objects maintai= ned by Flink state are live.

Flink keeps the state in memory even af= ter checkpointing , when does this state gets removed / GC. (I am using win= dow operator in which the DTO comes as input)

Also why d= oes Flink keep the state in memory after checkpointing ?=C2=A0

P.S U= sing RocksDB is not causing Full GC at all.

Regards,
Vinay Patil

=09 =09 =09 =09
=09
=09
=09
=09=09
If you reply to this email, your mess= age will be added to the discussion below:
=09=09http://apache-flink-us= er-mailing-list-archive.2336050.n4.nabble.com/Frequent-Full-GC-s-= in-case-of-FSStateBackend-tp11564p11565.html =09
=09
=09=09To start a new topic under Apache Flink User Mailing List archive., e= mail [hidden email]
=09=09To unsubscribe from Apache Flink User Mailing List archive., click here.
=09=09NAML<= /a> =09

=09 =09 =09
View this message in context: Re: Frequent Full GC's in case of FSStateBackend
Sent from the Ap= ache Flink User Mailing List archive. mailing list archive at Nabble.com.


=09 =09 =09 =09
=09
=09
=09
=09=09
If you reply to this email, your mess= age will be added to the discussion below:
=09=09http://apache-flink-us= er-mailing-list-archive.2336050.n4.nabble.com/Frequent-Full-= GC-s-in-case-of-FSStateBackend-tp11564p12126.html =09
=09
=09=09To start a new topic under Apache Flink User Mailing List archive., e= mail [hidden email]
=09=09To unsubscribe from Apache Flink User Mailing List archive., click here.<= br> =09=09NAML<= /a> =09
=09 =09 =09
View this message in context: Re: Frequent Full GC's in case of FSStateBackend
Sent from the Apache Flink User Mailing List archive. mailing list ar= chive at Nabble.com.
------=_Part_1848_1566169272.1489026079533--