flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Tinnemeier <ntinneme...@bol.com>
Subject Question about expired checkpoints
Date Fri, 16 Dec 2016 15:24:35 GMT
Hi all,

I am currently playing around with checkpoints to better understand how they work. I have
some questions I hope you can answer.
I am running a simple topology with a source, a map and a sink that writes the events it receives
to a HBase table. The parallelism of the environment is set to 10. Moreover, we set the parallelism
of the checkpoints to 20. The source is a custom one, implementing CheckpointListener interface.
In the notifyCheckpointComplete method of the source we simply log the checkpoint id of each
checkpoint that is being notified.

When I run the application, I notice that sometimes not all checkpoints are notified. Indeed,
when I look into the logs of the application I see messages like:
"Checkpoint 17 expired before completing"

I am sure the checkpoint did not timeout. What exactly does this mean and when does it happen?
It makes me wonder if this means that checkpoints can actually overtake each other. In other
words, can it actually happen that a checkpoint x arrives sooner at the sink than a checkpoint
x-1 that was sent earlier than x?

Kind regards,

View raw message