flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Cleanup of OperatorStates?
Date Tue, 01 Dec 2015 18:09:30 GMT

> On 01 Dec 2015, at 18:34, Stephan Ewen <sewen@apache.org> wrote:
> Hi!
> If you want to run with checkpoints (fault tolerance), you need to specify a place to
store the checkpoints to.
> By default, it is the master's memory (or zookeeper in HA), so we put a limit on the
size of the size of the state there.

Regarding the ZooKeeper in HA part: we don’t store the actual state in ZooKeeper, but a
pointer to the state (e.g. a pointer to the files, which in turn store the actual state).
So you don’t have to worry about ZooKeeper being flooded with your large data when you run
with HA.

– Ufuk

PS: Nice use case, indeed! :)
View raw message