flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangyu Su <xian...@smaato.com>
Subject Re: apache flink: Why checkpoint coordinator takes long time to get completion
Date Fri, 19 Jul 2019 10:21:07 GMT
Ok, thanks.

and this time-consuming until now always happens after 3rd checkpointing,
and this unexpected  time-consuming was always consistent (~ 4 min by under
4G/min incoming traffic).

On Fri, 19 Jul 2019 at 11:06, Biao Liu <mmyy1110@gmail.com> wrote:

> Hi Xiangyu,
>
> Just took a glance at the relevant codes. There is a gap between
> calculating the duration and logging it out. I guess the checkpoint 4 is
> finished in 1 minute, but there is an unexpected time-consuming operation
> during that time. But I can't tell which part it is.
>
>
> Xiangyu Su <xiangyu@smaato.com> 于2019年7月19日周五 下午4:14写道:
>
>> Dear flink community,
>>
>> We are POC flink(1.8) to process data in real time, and using global
>> checkpointing(S3) and local checkpointing(EBS), deploy cluster on EKS. Our
>> application is consuming data from Kinesis.
>>
>> For my test e.g I am using checkpointing interval 5min. and minimum pause
>> 2min.
>>
>> The issue what we saw is: It seems like flink checkpointing process would
>> be idle for 3-4 min, before job manager get complete notification.
>>
>> here is some logging from job manager:
>>
>> 2019-07-10 11:59:03,893 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator
    - Triggering checkpoint 4 @ 1562759941082 for job e7a97014f5799458f1c656135712813d.
>> 2019-07-10 12:05:01,836 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator
    - Completed checkpoint 4 for job e7a97014f5799458f1c656135712813d (22387207650 bytes in
58645 ms).
>>
>> As my understanding the logging above, the completedCheckpoint(CheckpointCoordinator)
>> object has been completed in 58645 ms, but the whole checkpointing process
>> took ~ 6min.
>>
>> This logging is for 4th checkpointing, But the first 3 checkpointing were
>> finished on time.
>> Could you please tell me, why flink checkpointing in my test was starting
>> "idle" for few minutes after 3 checkpointing?
>>
>> Best Regards
>> --
>> Xiangyu Su
>> Java Developer
>> xiangyu@smaato.com
>>
>> Smaato Inc.
>> San Francisco - New York - Hamburg - Singapore
>> www.smaato.com
>>
>> Germany:
>> Valentinskamp 70, Emporio, 19th Floor
>> 20355 Hamburg
>> M 0049(176)22943076
>>
>> The information contained in this communication may be CONFIDENTIAL and
>> is intended only for the use of the recipient(s) named above. If you are
>> not the intended recipient, you are hereby notified that any dissemination,
>> distribution, or copying of this communication, or any of its contents, is
>> strictly prohibited. If you have received this communication in error,
>> please notify the sender and delete/destroy the original message and any
>> copy of it from your computer or paper files.
>>
>

-- 
Xiangyu Su
Java Developer
xiangyu@smaato.com

Smaato Inc.
San Francisco - New York - Hamburg - Singapore
www.smaato.com

Germany:
Valentinskamp 70, Emporio, 19th Floor
20355 Hamburg
M 0049(176)22943076

The information contained in this communication may be CONFIDENTIAL and is
intended only for the use of the recipient(s) named above. If you are not
the intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error,
please notify the sender and delete/destroy the original message and any
copy of it from your computer or paper files.

Mime
View raw message