flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Question on checkpointing
Date Mon, 01 May 2017 18:03:53 GMT
Hi Blake,
I didn’t see any mails on the user mailing list regarding this. Is it still relevant.

From the looks of it you might be interested in the savepoints feature: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html>.

This uses the checkpointing mechanism to savepoint the state of a running job so that the
job can be canceled and later restored.

Best,
Aljoscha 
> On 21. Apr 2017, at 20:42, Blake Essing <essings@swbell.net> wrote:
> 
> Sorry all for my earlier messages.  I just realized that I'm on the wrong mailing list
for what I'm asking.  I will move my question to the user list. 
> 
>    On Friday, April 21, 2017 11:54 AM, Blake Essing <essings@swbell.net> wrote:
> 
> 
> Sorry for the earlier incomplete email, E-mail editor issue.
> I have modified the WordCounter example program to use a NiFi DataStream as input.  I
would like to add checkpointing to the program to learn how it works.  I have googled and
added many of the checkpointing configuration values to the StreamExecutionEnvironment but
I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing
should work.  My goal is to save state so that if the program is stopped and then restarted
the counts will continue from the last checkpoint.
> The counts variable is populated by
> 
> DataStream<WordCounter> counts = text    .flatMap(new Tokenizer())    .keyBy("word")
   .reduce(new ReduceFunction<WordCounter>() {        public WordCounter reduce(WordCounter
a, WordCounter b) throws Exception {            return new WordCounter(a.getWord, a.getCount()
+ b.getCount());        }    })
> text is just a  DataStream<String> populated from NiFi where lines are broken up
into individual strings.
> WordCounter is a simple pojo that stores the word and a count with get/set methods.
> Please let me know if you have any questions about the code. 
> 
>     On Friday, April 21, 2017 11:48 AM, Blake Essing <essings@swbell.net> wrote:
> 
> 
> I have modified the WordCounter example program to use a NiFi DataStream as input.  I
would like to add checkpointing to the program to learn how it works.  I have googled and
added many of the checkpointing configuration values to the StreamExecutionEnvironment but
I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing
should work.  My goal is to save state so that if the program is stopped and then restarted
the counts will continue from the last checkpoint.  The counts variable is populated by
> 
>   
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message