flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rinat <r.shari...@cleverdata.ru>
Subject Re: Flink send checkpointing message in IT
Date Thu, 02 Nov 2017 18:11:40 GMT
Chesnay, thanks for your reply, it was very helpful, but I took logic from this test template
and tried to reuse it in my IT case, but found one more issue.
I’ve registered an accumulator in my source function, and for it’s value, as specified
in the specified example.
When accumulator has an expected value, I perform a savepoint and wait for it’s completion
using the further code

ActorGateway jobManager = (ActorGateway) Await.result(cluster.leaderGateway().future(), DEADLINE.timeLeft());
Future<Object> savepointResultFuture = jobManager.ask(new JobManagerMessages.TriggerSavepoint(
    jobId, Option.<String>empty()), DEADLINE.timeLeft()
Object savepointResult = Await.result(savepointResultFuture, DEADLINE.timeLeft());
Afterwards, if failures haven’t been detected I cancels my job and shutdowns cluster.

I found, that checkpoint method notifyCheckpointComplete not always called, before the savepointResult
is ready. So the part of my logic, that lives in implementation of this method doesn’t work
and test fails.

So could you or someone explain, does Flink guaranties, that notifyCheckpointComplete method
will be called before savepointResult  will become accessable.
For me, it’s rather strange behaviour and I think that I’m doing something wrong.


> On 1 Nov 2017, at 14:26, Chesnay Schepler <chesnay@apache.org> wrote:
> You could trigger a savepoint, which from the viewpoint of sources/operators/sinks is
the same thing as a checkpoint.
> How to do this depends a bit on how your test case is written, but you can take a look
at the SavepointMigrationTestBase#executeAndSavepoint which is all about running josb and
> savepoints once certain conditions have been met.
> On 30.10.2017 16:01, Rinat wrote:
>> Hi guys, I’ve got a question about working with checkpointing.
>> I would like to implement IT test, where source is a fixed collection of items and
sink performs additional logic, when checkpointing is completed.
>> I would like to force executing checkpointing, when all messages from my test source
were sent and processed by sink.
>> Please tell me, whether such logic could be performed or not, and how.
>> Thx !

View raw message