flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Issues in recovering state from last crash using custom sink
Date Mon, 28 Aug 2017 09:52:04 GMT
Hi,

How are you testing the recovery behaviour? Are you taking a savepoint ,then shutting down,
and then restarting the Job from the savepoint?

Best,
Aljoscha

> On 28. Aug 2017, at 00:28, vipul singh <neoeahit@gmail.com> wrote:
> 
> Hi all,
> 
> I am working on a flink archiver application. In a gist this application tries to reads
a bunch of schematized messages from kafka and archives them to s3. Due to the nature of the
naming of the files, I had to go towards a custom sink implementation. As of the current progress
the application in general is able to archive files to s3 ok.
> I am having some issues during the recovery phase. A sample of the code can be found
on link <https://gist.github.com/neoeahit/4c80af355d5166eab07fbae1079060fe>. My issue
is on recovery when initializeState is called, it is not able to get(recover) the last checkpointed
ListState( i.e. checkpointedMessages <https://gist.github.com/neoeahit/4c80af355d5166eab07fbae1079060fe#file-gistfile1-txt-L47>
is 0). I think this might be because of the way I am retrieving the checkpointed messages.
Could someone please point me to what is wrong? or direct me to some examples which do a similar
thing( Please note Message <https://gist.github.com/neoeahit/4c80af355d5166eab07fbae1079060fe#file-gistfile1-txt-L2>
class is our own implementation)
> 
> Thanks,
> Vipul


Mime
View raw message