flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: custom sources
Date Fri, 20 May 2016 10:58:21 GMT
On Thu, May 19, 2016 at 7:48 PM, Abhishek R. Singh
<abhishsi@tetrationanalytics.com> wrote:
> There seems to be some relationship between watermarks, triggers and
> checkpoint that is someone not being leveraged.

Checkpointing is independent of this, yes. Did the state size become a
problem for your use case? There are various users running Flink with
very large state sizes without any issues. The recommended state
backend for these use cases is the RocksDB backend.

The barriers are triggered at the sources and flow with the data
(https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/stream_checkpointing.html).
Everything in-flight after the barrier is not relevant for the
checkpoint. We are only interested in a consistent state snapshot.

Mime
View raw message