flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Lublinsky <boris.lublin...@lightbend.com>
Subject Re: HA and zookeeper
Date Mon, 08 Apr 2019 13:20:04 GMT
Thanks.
Is there:
1.  Documentation, describing this?
2. Any proposals/work trying to store it elsewhere?

The reason for this question is kubernetes deployment, where the use of zookeeper seems an
overkill, but it will not work without zookeeper, see https://jobs.zalando.com/tech/blog/running-apache-flink-on-kubernetes/?gh_src=4n3gxh1
<https://jobs.zalando.com/tech/blog/running-apache-flink-on-kubernetes/?gh_src=4n3gxh1>

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

> On Apr 8, 2019, at 10:29 AM, Fabian Hueske <fhueske@gmail.com> wrote:
> 
> Hi Boris,
> 
> ZooKeeper is also used by the JobManager to store metadata about the running job.
> The JM writes information like the JobGraph, JAR file, checkpoint metadata to a persistent
storage (like HDFS, S3, ...) and a pointer to this information to ZooKeeper.
> In case of a recovery, the new JM looks up the pointer from ZooKeeper and fetches the
job metadata from the persistent storage.
> 
> Best, Fabian
> 
> Am Sa., 6. Apr. 2019 um 01:28 Uhr schrieb Boris Lublinsky <boris.lublinsky@lightbend.com
<mailto:boris.lublinsky@lightbend.com>>:
> For HA implementation, is zookeeper is used only for leader selection, or it also stores
some data relevant for switching to backup server
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <mailto:boris.lublinsky@lightbend.com>
> https://www.lightbend.com/ <https://www.lightbend.com/>


Mime
View raw message