flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bart van Deenen <bartvandee...@fastmail.fm>
Subject mutable hashmap outside of stream, does it get snapshotted ?
Date Thu, 07 Apr 2016 13:41:06 GMT
Hi all

I'm having a datastream transformation, that updates a mutable 
hashmap that exists outside of the stream.

So it's something like

object FlinkJob {
  val uriLookup = mutable.HashMap.empty[String, Int]


  def main(args: Array[String]) {

    val stream: DataStream = ...

    stream.keybBy(1).timeWindow(..).fold(..)
    .window(..)
    .map(..).fold(..)
    .addSink(..)
  }
}

where the uriLookup hashmap gets updated inside the stream
transformation, 
and is serialized in the step before the addSink

It works fine, however

Does the snapshotting mechanism in case of a node failure actually
serialize this map?

And out of curiousity, can I actually see what data exists inside the
snapshot data?

Thanks.

Bart


Mime
View raw message