flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay patil <vinay18.pa...@gmail.com>
Subject Checkpointing with RocksDB as statebackend
Date Wed, 15 Feb 2017 11:11:11 GMT

I have kept the checkpointing interval to 6secs and minimum pause between
checkpoints to 5secs, while testing the pipeline I have observed that that
for some checkpoints it is taking long time , as you can see in the attached
snapshot checkpoint id 19 took the maximum time before it gets failed,
although it has not received any acknowledgements, now during this 10minutes
the entire pipeline did not make any progress and no data was getting
processed. (For Ex : In 13minutes 20M records were processed and when the
checkpoint took time there was no progress for the next 10minutes)

I have even tried to set max checkpoint timeout to 3min, but in that case as
well multiple checkpoints were getting failed.

I have set RocksDB FLASH_SSD_OPTION 
What could be the issue ? 

P.S. I am writing to 3 S3 sinks 


View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message