flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Zagrebin <and...@data-artisans.com>
Subject Re: Aggregator State in Keyed Windowed Stream
Date Mon, 10 Sep 2018 13:40:47 GMT
Hi Ning,

> Back to my first question, is the accumulator state backed by RocksDB state backend?
If so, I don’t need to use rich function for the aggregate function.

the answer is yes, it is backed by state backend (should be RocksDB if you configure it),

you can trace it through these method calls:

sourceStream.keyBy(…)
    .timeWindow(Time.seconds(…))
    .trigger(CountTrigger.of(…))
gives you WindowedStream,
WindowedStream.aggregate(new MyAggFunc()) creates:
new WindowOperator(windowStateDescriptor = new AggregatingStateDescriptor()), 
inside WindowOperator: 
WindowOperator.open() uses configured backend to create windowState,
WindowOperator.processElement() uses windowState which is AggregatingState.

Cheers,
Andrey

> On 10 Sep 2018, at 13:39, Ning Shi <ningshi2@gmail.com> wrote:
> 
> Hi Vino,
> 
>> If you need access to the state API, you can consider using ProcessWindowFunction[1],
which allows you to use ProcessWindowFunction.
> 
> I was hoping that I could use the aggregate function to do incremental aggregation. My
understanding is that ProcessWindowFunction either has to loop through all records or be combined
with an aggregate function to do incremental aggregation.
> 
> Back to my first question, is the accumulator state backed by RocksDB state backend?
If so, I don’t need to use rich function for the aggregate function.
> 
> Thanks,
> 
> Ning


Mime
View raw message