flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vino yang <yanghua1...@gmail.com>
Subject Re: Aggregator State in Keyed Windowed Stream
Date Mon, 10 Sep 2018 07:20:46 GMT
In addition :

ProcessWindowFunction extends AbstractRichFunction, through
getRuntimeContext,you can access keyed state API.

vino yang <yanghua1127@gmail.com> 于2018年9月10日周一 下午3:19写道:

> Hi Ning,
>
> Answer you question:
>
> *And why is rich functions not allowed here?*
>
> If you need access to the state API, you can consider using
> ProcessWindowFunction[1], which allows you to use ProcessWindowFunction.
>
> Thanks, vino.
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>
> Ning Shi <ningshi2@gmail.com> 于2018年9月10日周一 上午11:28写道:
>
>> Since the aggregate() function on a keyed window stream does not allow
>> using rich functions, I can only use an AggregateFunction. Is the
>> accumulator state of the AggregateFunction backed by RocksDB and
>> persisted in checkpoints if I use the RocksDB backend. My job looks
>> like the following,
>>
>> sourceStream.keyBy(…)
>>     .timeWindow(Time.seconds(…))
>>     .trigger(CountTrigger.of(…))
>>     .aggregate(new MyAggFunc());
>>
>> Since the stream has high cardinality keys, each window could have
>> millions of them. If the accumulator state is not backed by RocksDB,
>> there might be a lot of data stored on the heap.
>>
>> And why is rich functions not allowed here?
>>
>> Thanks,
>>
>> —
>> Ning
>>
>

Mime
View raw message