flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: JobManager HA without Distributed FileSystem
Date Tue, 23 Aug 2016 18:45:41 GMT
Hi!

The state one can store in ZooKeeper is only very small (recommended is
smaller than 1MB per handle).

For HA, the JobManager needs to persist:
  - JobGraph
  - JAR files
  - Checkpoint Metadata

Those are easily too large for ZooKeeper, which is why Flink currently
requires a DFS to store those, and only stores "pointers" to the data in
the DFS in ZooKeeper.

Are you thinking of another highly available storage for larger data
(megabytes) that could be used here?

Greetings,
Stephan


On Tue, Aug 23, 2016 at 6:36 PM, Konstantin Knauf <
konstantin.knauf@tngtech.com> wrote:

> Hi all,
>
> the documenation of JobManager HA [1] explains that HA is only possible
> with the FS state backend as Job Manager metadata is saved there.
>
> What are the particular problems using JobManager HA with the
> MemoryStatebackend?
>
> As I understand it, the state is checkpointed to all JobManagers
> (leaders + standy) when using the MemoryStateBackend or am I wrong here?
>
> Follow Up Question: Is it generally possible to setup a highly
> available, at-least-once (source: Kafka) pipeline without a distributed
> filesystem (only local FS and Zookeeper) for the checkpoints?
>
> Cheers,
>
> Konstantin
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-
> master/setup/jobmanager_high_availability.html
>
> --
> Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Mime
View raw message