flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Blake Essing <essi...@swbell.net>
Subject Re: Question on checkpointing
Date Fri, 21 Apr 2017 16:54:15 GMT
Sorry for the earlier incomplete email, E-mail editor issue.
I have modified the WordCounter example program to use a NiFi DataStream as input.  I would
like to add checkpointing to the program to learn how it works.  I have googled and added
many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing
something critical to getting this to work or I'm misinterpreting how the checkpointing should
work.  My goal is to save state so that if the program is stopped and then restarted the
counts will continue from the last checkpoint.
The counts variable is populated by

DataStream<WordCounter> counts = text    .flatMap(new Tokenizer())    .keyBy("word") 
  .reduce(new ReduceFunction<WordCounter>() {        public WordCounter reduce(WordCounter
a, WordCounter b) throws Exception {            return new WordCounter(a.getWord, a.getCount()
+ b.getCount());        }    })
text is just a  DataStream<String> populated from NiFi where lines are broken up into
individual strings.
WordCounter is a simple pojo that stores the word and a count with get/set methods.
Please let me know if you have any questions about the code. 

    On Friday, April 21, 2017 11:48 AM, Blake Essing <essings@swbell.net> wrote:
 

 I have modified the WordCounter example program to use a NiFi DataStream as input.  I would
like to add checkpointing to the program to learn how it works.  I have googled and added
many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing
something critical to getting this to work or I'm misinterpreting how the checkpointing should
work.  My goal is to save state so that if the program is stopped and then restarted the
counts will continue from the last checkpoint.  The counts variable is populated by

   
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message