flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hironori Ogibayashi <ogibaya...@gmail.com>
Subject Handling large state (incremental snapshot?)
Date Tue, 05 Apr 2016 12:05:56 GMT
Hello,

I am trying to implement windowed distinct count on a stream. In this
case, the state
have to hold all distinct value in the window, so can be large.

In my test, if the state size become about 400MB, checkpointing takes
40sec and spends most of Taskmanager's CPU.
Are there any good way to handle this situation?

Flink document mentions about incremental snapshot, and I am interested in it,
but could not find how to enable it. (not implemented yet?)
https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/stream_checkpointing.html

Regards,
Hironori

Mime
View raw message