flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Checkpointing with RocksDB as statebackend
Date Mon, 20 Feb 2017 18:51:50 GMT
Hi Vinay!

Can you start by giving us a bit of an environment spec?

  - What Flink version are you using?
  - What is your rough topology (what operations does the program use)
  - Where is the state (windows, keyBy)?
  - What is the rough size of your checkpoints and where does the time go?
Can you attach a screenshot from
https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/checkpoint_monitoring.html
  - What is the size of the JVM?

Those things would be helpful to know...

Best,
Stephan


On Mon, Feb 20, 2017 at 7:04 PM, vinay patil <vinay18.patil@gmail.com>
wrote:

> Hi Xiaogang,
>
> Thank you for your inputs.
>
> Yes I have already tried setting MaxBackgroundFlushes and
> MaxBackgroundCompactions to higher value (tried with 2, 4, 8) , still not
> getting expected results.
>
> System.getProperty("java.io.tmpdir") points to /tmp but there I could not
> find RocksDB logs, can you please let me know where can I find it ?
>
> Regards,
> Vinay Patil
>
> On Mon, Feb 20, 2017 at 7:32 AM, xiaogang.sxg [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=11752&i=0>> wrote:
>
>> Hi Vinay
>>
>> Can you provide the LOG file in RocksDB? It helps a lot to figure out
>> the problems becuse it records the options and the events happened
>> during the execution. Otherwise configured, it should locate at the path
>> set in System.getProperty("java.io.tmpdir").
>>
>> Typically, a large amount of memory is consumed by RocksDB to store
>> necessary indices. To avoid the unlimited growth in the memory consumption,
>> you can put these indices into block cache (set CacheIndexAndFilterBlock to
>> true) and properly set the block cache size.
>>
>> You can also increase the number of backgroud threads to improve the
>> performance of flushes and compactions (via MaxBackgroundFlushes and
>> MaxBackgroudCompactions).
>>
>> In YARN clusters, task managers will be killed if their memory
>> utilization exceeds the allocation size. Currently Flink does not count the
>> memory used by RocksDB in the allocation. We are working on fine-grained
>> resource allocation (see FLINK-5131). It may help to avoid such problems.
>>
>> May the information helps you.
>>
>> Regards,
>> Xiaogang
>>
>>
>> ------------------------------------------------------------------
>> 发件人:Vinay Patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11731&i=0>>
>> 发送时间:2017年2月17日(星期五) 21:19
>> 收件人:user <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11731&i=1>>
>> 主 题:Re: Checkpointing with RocksDB as statebackend
>>
>> Hi Guys,
>>
>> There seems to be some issue with RocksDB memory utilization.
>>
>> Within few minutes of job run the physical memory usage increases by 4-5
>> GB and it keeps on increasing.
>> I have tried different options for Max Buffer Size(30MB, 64MB, 128MB ,
>> 512MB) and Min Buffer to Merge as 2, but the physical memory keeps on
>> increasing.
>>
>> According to RocksDB documentation, these are the main options on which
>> flushing to storage is based.
>>
>> Can you please point me where am I doing wrong. I have tried different
>> configuration options but each time the Task Manager is getting killed
>> after some time :)
>>
>> Regards,
>> Vinay Patil
>>
>> On Thu, Feb 16, 2017 at 6:02 PM, Vinay Patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11731&i=2>> wrote:
>> I think its more of related to RocksDB, I am also not aware about RocksDB
>> but reading the tuning guide to understand the important values that can be
>> set
>>
>> Regards,
>> Vinay Patil
>>
>> On Thu, Feb 16, 2017 at 5:48 PM, Stefan Richter [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11731&i=3>> wrote:
>> What kind of problem are we talking about? S3 related or RocksDB related.
>> I am not aware of problems with RocksDB per se. I think seeing logs for
>> this would be very helpful.
>>
>> Am 16.02.2017 um 11:56 schrieb Aljoscha Krettek <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11673&i=0>>:
>>
>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=11673&i=1>
and [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=11673&i=2> could
this
>> be the same problem that you recently saw when working with other people?
>>
>> On Wed, 15 Feb 2017 at 17:23 Vinay Patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11673&i=3>> wrote:
>> Hi Guys,
>>
>> Can anyone please help me with this issue
>>
>> Regards,
>> Vinay Patil
>>
>> On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11673&i=4>> wrote:
>> Hi Ted,
>>
>> I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3
>> sink and the 3rd box is window operator followed by chained operators and a
>> s3 sink
>>
>> So in the details link section I can see that that S3 sink is taking time
>> for the acknowledgement and it is not even going to the window operator
>> chain.
>>
>> But as shown in the snapshot ,checkpoint id 19 did not get any
>> acknowledgement. Not sure what is causing the issue
>>
>> Regards,
>> Vinay Patil
>>
>> On Wed, Feb 15, 2017 at 5:51 PM, Ted Yu [via Apache Flink User Mailing
>> List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11673&i=5>> wrote:
>> What did the More Details link say ?
>>
>> Thanks
>>
>> > On Feb 15, 2017, at 3:11 AM, vinay patil <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=11641&i=0>> wrote:
>> >
>> > Hi,
>> >
>> > I have kept the checkpointing interval to 6secs and minimum pause
>> between
>> > checkpoints to 5secs, while testing the pipeline I have observed that
>> that
>> > for some checkpoints it is taking long time , as you can see in the
>> attached
>> > snapshot checkpoint id 19 took the maximum time before it gets failed,
>> > although it has not received any acknowledgements, now during this
>> 10minutes
>> > the entire pipeline did not make any progress and no data was getting
>> > processed. (For Ex : In 13minutes 20M records were processed and when
>> the
>> > checkpoint took time there was no progress for the next 10minutes)
>> >
>> > I have even tried to set max checkpoint timeout to 3min, but in that
>> case as
>> > well multiple checkpoints were getting failed.
>> >
>> > I have set RocksDB FLASH_SSD_OPTION
>> > What could be the issue ?
>> >
>> > P.S. I am writing to 3 S3 sinks
>> >
>> > checkpointing_issue.PNG
>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/file/n11640/checkpointing_issue.PNG>
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/Checkpointing-with-
>> RocksDB-as-statebackend-tp11640.html
>> > Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11641.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=11673&i=6>
>> To unsubscribe from Apache Flink User Mailing List archive., click here
>> <#m_6198963695418156302_m_8892162958879126193_this>.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11673.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=11731&i=4>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.
> nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11731.html
> To start a new topic under Apache Flink User Mailing List archive., email [hidden
> email] <http:///user/SendEmail.jtp?type=node&node=11752&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> ------------------------------
> View this message in context: Re: Checkpointing with RocksDB as
> statebackend
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752.html>
>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>

Mime
View raw message