flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Checkpointing with RocksDB as statebackend
Date Thu, 23 Feb 2017 20:12:09 GMT
The FSStatebackend uses the heap to keep the data, only the state snapshots
are stored in the file system.

On Thu, Feb 23, 2017 at 6:13 PM, vinay patil <vinay18.patil@gmail.com>
wrote:

> Hi,
>
> When I disabled checkpointing the memory usage is similar for all nodes,
> this means that for  checkpointing enabled case  the data is first flushed
> to memory of CORE nodes (DataNode daemon is running here in case of EMR ) .
>
> I am going to run with FSStatebackend on a high end cluster with 122GB
> RAM, in case of FSStatebackend does it use TM heap memory or physical
> memory to store the state ?
>
> Regards,
> Vinay Patil
>
> On Thu, Feb 23, 2017 at 7:50 PM, vinay patil [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=11845&i=0>> wrote:
>
>> Hi Stephan,
>>
>> Anyways the Async exception is gone.
>>
>> I have increased my instance type to r3.2xlarge having 60GB of memory.
>> BUt what I have observed here is that for two task managers the memory
>> usage is close to 30GB but for other two it goes up to 55GB, the load is
>> equally distributed among all TM's.
>> Why does this happen ?
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-
>> tp11752p11831.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=11845&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Checkpointing with RocksDB as
> statebackend
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11845.html>
>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>

Mime
View raw message