flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Radford <mub...@gmail.com>
Subject FoldFunction accumulator checkpointing
Date Tue, 12 Apr 2016 22:29:14 GMT
I'm wondering whether the accumulator value maintained by a
FoldFunction is automatically checkpointed?

In general, but specifically when using the WindowedStream.apply
variant that takes a FoldFunction:

public <R> DataStream<R> apply(R initialValue,
                      FoldFunction<T,R> foldFunction,
                      WindowFunction<R,R,K,W> function,
                      TypeInformation<R> evidence$7)

If not, then Flink 1.0.1 still has the issue that you can't pass a
RichFoldFunction to WindowedStream.apply
(java.lang.UnsupportedOperationException: ReduceFunction of apply can
not be a RichFunction).

But also, if not, it seems like this would be a common pattern when
doing complex (keyed / multi-valued) aggregations, and if the
accumulator type R is serializable, there could be a convenience
method for a checkpointed fold, like the mapWithState mentioned in the
State section of the streaming guide.

Thanks,
Mike

Mime
View raw message