flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Shi <nings...@gmail.com>
Subject Aggregator State in Keyed Windowed Stream
Date Mon, 10 Sep 2018 03:28:06 GMT
Since the aggregate() function on a keyed window stream does not allow
using rich functions, I can only use an AggregateFunction. Is the
accumulator state of the AggregateFunction backed by RocksDB and
persisted in checkpoints if I use the RocksDB backend. My job looks
like the following,

sourceStream.keyBy(…)
    .timeWindow(Time.seconds(…))
    .trigger(CountTrigger.of(…))
    .aggregate(new MyAggFunc());

Since the stream has high cardinality keys, each window could have
millions of them. If the accumulator state is not backed by RocksDB,
there might be a lot of data stored on the heap.

And why is rich functions not allowed here?

Thanks,

—
Ning

Mime
View raw message