flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: RollingSink
Date Tue, 22 Mar 2016 10:04:08 GMT
how are you printing the debug statements?

But yeah all the logic of renaming in progress files and cleaning up after a failed job happens
in restoreState(BucketState state). The steps are roughly these:

1. Move current in-progress file to final location
2. truncate the file if necessary (if truncate is not available write a .valid-length file)
3. Move pending files to final location that where part of the checkpoint
4. cleanup any leftover pending/in-progress files

> On 22 Mar 2016, at 10:08, Vijay Srinivasaraghavan <vijikarthi@yahoo.com.INVALID>
> Hello,
> I have enabled checkpoint and I am using RollingSink to sink the data to HDFS (2.7.x)
from KafkaConsumer. To simulate failover/recovery, I stopped TaskManager and the job gets
rescheduled to other Taskmanager instance. During this momemnt, the current "in-progress"
gets closed and renamed to part-0-1 from _part-0-1_in-progress. 
> I was hoping to see the debug statement that I have added to "restoreState" method but
none of my debug statement gets printed. I am not sure if the restoreState() method gets invoked
during this scenario. Could you please help me understand the flow during "failover" scenario?
> P.S: Functionally the code appears to be working fine but I am trying to understand the
underlying implementation details. public void restoreState(BucketState state)
> Regards
> Vijay

View raw message