flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Checkpointing with RocksDB as statebackend
Date Mon, 20 Feb 2017 19:25:45 GMT
@Vinay!

Just saw the screenshot you attached to the first mail. The checkpoint that
failed came after one that had an incredible heavy alignment phase (14 GB).
I think that working that off threw the next checkpoint because the workers
were still working off the alignment backlog.

I think you can for now fix this by setting the minimum pause between
checkpoints a bit higher (it is probably set a bit too small for the state
of your application).

Also, can you describe what your sources are (Kafka / Kinesis or file
system)?

BTW: We are currently working on
  - incremental RocksDB checkpoints
  - the network stack to allow in the future for a new way of doing the
alignment

Both of that should help that the program is more resilient to these
situations.

Best,
Stephan



On Mon, Feb 20, 2017 at 7:51 PM, Stephan Ewen <sewen@apache.org> wrote:

> Hi Vinay!
>
> Can you start by giving us a bit of an environment spec?
>
>   - What Flink version are you using?
>   - What is your rough topology (what operations does the program use)
>   - Where is the state (windows, keyBy)?
>   - What is the rough size of your checkpoints and where does the time go?
> Can you attach a screenshot from https://ci.apache.org/
> projects/flink/flink-docs-release-1.2/monitoring/
> checkpoint_monitoring.html
>   - What is the size of the JVM?
>
> Those things would be helpful to know...
>
> Best,
> Stephan
>
>
> On Mon, Feb 20, 2017 at 7:04 PM, vinay patil <vinay18.patil@gmail.com>
> wrote:
>
>> Hi Xiaogang,
>>
>> Thank you for your inputs.
>>
>> Yes I have already tried setting MaxBackgroundFlushes and
>> MaxBackgroundCompactions to higher value (tried with 2, 4, 8) , still not
>> getting expected results.
>>
>> System.getProperty("java.io.tmpdir") points to /tmp but there I could
>> not find RocksDB logs, can you please let me know where can I find it ?
>>
>> Regards,
>> Vinay Patil
>>
>> On Mon, Feb 20, 2017 at 7:32 AM, xiaogang.sxg [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11752&i=0>> wrote:
>>
>>> Hi Vinay
>>>
>>> Can you provide the LOG file in RocksDB? It helps a lot to figure out
>>> the problems becuse it records the options and the events happened
>>> during the execution. Otherwise configured, it should locate at the
>>> path set in System.getProperty("java.io.tmpdir").
>>>
>>> Typically, a large amount of memory is consumed by RocksDB to store
>>> necessary indices. To avoid the unlimited growth in the memory consumption,
>>> you can put these indices into block cache (set CacheIndexAndFilterBlock to
>>> true) and properly set the block cache size.
>>>
>>> You can also increase the number of backgroud threads to improve the
>>> performance of flushes and compactions (via MaxBackgroundFlushes and
>>> MaxBackgroudCompactions).
>>>
>>> In YARN clusters, task managers will be killed if their memory
>>> utilization exceeds the allocation size. Currently Flink does not count the
>>> memory used by RocksDB in the allocation. We are working on fine-grained
>>> resource allocation (see FLINK-5131). It may help to avoid such problems.
>>>
>>> May the information helps you.
>>>
>>> Regards,
>>> Xiaogang
>>>
>>>
>>> ------------------------------------------------------------------
>>> 发件人:Vinay Patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11731&i=0>>
>>> 发送时间:2017年2月17日(星期五) 21:19
>>> 收件人:user <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11731&i=1>>
>>> 主 题:Re: Checkpointing with RocksDB as statebackend
>>>
>>> Hi Guys,
>>>
>>> There seems to be some issue with RocksDB memory utilization.
>>>
>>> Within few minutes of job run the physical memory usage increases by 4-5
>>> GB and it keeps on increasing.
>>> I have tried different options for Max Buffer Size(30MB, 64MB, 128MB ,
>>> 512MB) and Min Buffer to Merge as 2, but the physical memory keeps on
>>> increasing.
>>>
>>> According to RocksDB documentation, these are the main options on which
>>> flushing to storage is based.
>>>
>>> Can you please point me where am I doing wrong. I have tried different
>>> configuration options but each time the Task Manager is getting killed
>>> after some time :)
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Thu, Feb 16, 2017 at 6:02 PM, Vinay Patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11731&i=2>> wrote:
>>> I think its more of related to RocksDB, I am also not aware about
>>> RocksDB but reading the tuning guide to understand the important values
>>> that can be set
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Thu, Feb 16, 2017 at 5:48 PM, Stefan Richter [via Apache Flink User
>>> Mailing List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11731&i=3>> wrote:
>>> What kind of problem are we talking about? S3 related or RocksDB
>>> related. I am not aware of problems with RocksDB per se. I think seeing
>>> logs for this would be very helpful.
>>>
>>> Am 16.02.2017 um 11:56 schrieb Aljoscha Krettek <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=0>>:
>>>
>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=11673&i=1>
>>>  and [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=2> could this
be the
>>> same problem that you recently saw when working with other people?
>>>
>>> On Wed, 15 Feb 2017 at 17:23 Vinay Patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=3>> wrote:
>>> Hi Guys,
>>>
>>> Can anyone please help me with this issue
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=4>> wrote:
>>> Hi Ted,
>>>
>>> I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3
>>> sink and the 3rd box is window operator followed by chained operators and a
>>> s3 sink
>>>
>>> So in the details link section I can see that that S3 sink is taking
>>> time for the acknowledgement and it is not even going to the window
>>> operator chain.
>>>
>>> But as shown in the snapshot ,checkpoint id 19 did not get any
>>> acknowledgement. Not sure what is causing the issue
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Wed, Feb 15, 2017 at 5:51 PM, Ted Yu [via Apache Flink User Mailing
>>> List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=5>> wrote:
>>> What did the More Details link say ?
>>>
>>> Thanks
>>>
>>> > On Feb 15, 2017, at 3:11 AM, vinay patil <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=11641&i=0>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have kept the checkpointing interval to 6secs and minimum pause
>>> between
>>> > checkpoints to 5secs, while testing the pipeline I have observed that
>>> that
>>> > for some checkpoints it is taking long time , as you can see in the
>>> attached
>>> > snapshot checkpoint id 19 took the maximum time before it gets failed,
>>> > although it has not received any acknowledgements, now during this
>>> 10minutes
>>> > the entire pipeline did not make any progress and no data was getting
>>> > processed. (For Ex : In 13minutes 20M records were processed and when
>>> the
>>> > checkpoint took time there was no progress for the next 10minutes)
>>> >
>>> > I have even tried to set max checkpoint timeout to 3min, but in that
>>> case as
>>> > well multiple checkpoints were getting failed.
>>> >
>>> > I have set RocksDB FLASH_SSD_OPTION
>>> > What could be the issue ?
>>> >
>>> > P.S. I am writing to 3 S3 sinks
>>> >
>>> > checkpointing_issue.PNG
>>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.na
>>> bble.com/file/n11640/checkpointing_issue.PNG>
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context: http://apache-flink-user-maili
>>> ng-list-archive.2336050.n4.nabble.com/Checkpointing-with-Roc
>>> ksDB-as-statebackend-tp11640.html
>>> > Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11641.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11673&i=6>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>> <#m_3724869264661144930_m_6198963695418156302_m_8892162958879126193_this>
>>> .
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11673.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=11731&i=4>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>> ble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11731.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=11752&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> ------------------------------
>> View this message in context: Re: Checkpointing with RocksDB as
>> statebackend
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752.html>
>>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>
>

Mime
View raw message