flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: FoldFunction accumulator checkpointing
Date Wed, 13 Apr 2016 08:25:12 GMT
Hi,
there are two cases where a FoldFunction can be used in the streaming API:
KeyedStream.fold() and WindowedStream.fold()/apply(). In both cases we
internally use the partitioned state abstraction of Flink to keep the
state. So yes, the accumulator value is consistently maintained and will
survive failures.

Right now, the accumulation function of a window cannot be a rich function
because the underlying state primitives that the windowing system uses can
only take plain functions because supporting rich functions there could
have problematic implications. The most obvious one to me seems that users
could be trying to keep state in the ReduceFunction of a ReducingState when
given the chance to do so, which a RichFunction does.

This is just off the top of my head but I can go into detail if you want.

Cheers,
Aljoscha

On Wed, 13 Apr 2016 at 00:29 Michael Radford <mubber@gmail.com> wrote:

> I'm wondering whether the accumulator value maintained by a
> FoldFunction is automatically checkpointed?
>
> In general, but specifically when using the WindowedStream.apply
> variant that takes a FoldFunction:
>
> public <R> DataStream<R> apply(R initialValue,
>                       FoldFunction<T,R> foldFunction,
>                       WindowFunction<R,R,K,W> function,
>                       TypeInformation<R> evidence$7)
>
> If not, then Flink 1.0.1 still has the issue that you can't pass a
> RichFoldFunction to WindowedStream.apply
> (java.lang.UnsupportedOperationException: ReduceFunction of apply can
> not be a RichFunction).
>
> But also, if not, it seems like this would be a common pattern when
> doing complex (keyed / multi-valued) aggregations, and if the
> accumulator type R is serializable, there could be a convenience
> method for a checkpointed fold, like the mapWithState mentioned in the
> State section of the streaming guide.
>
> Thanks,
> Mike
>

Mime
View raw message