flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: quesiont related to store location of save point when using file:///, instead of s3 and hdfs
Date Tue, 28 Mar 2017 16:47:08 GMT

if you use the MemoryStateBackend, the state is shipped from the TMs to the
JM. The JM holds the state in its own memory (i.e., on the JVM heap).
This means that the MemoryStateBackend is not reliable by design and rather
meant for testing and running Flink in your IDE.

If you want to persist the state in a distributed filesystem, you should
use the FSStateBackend. Similar to the MemoryStateBackend, the
FSStateBackend stores the data on the TM's heap, but on a checkpoint the
state is written into a configurable FS directory.

Best, Fabian

2017-03-28 18:00 GMT+02:00 Sendoh <unicorn.banachi@gmail.com>:

> Hi Flink users,
> When using `file:///mysavepoint` instead of `hdsf://mysavepoint  for
> ENVCONFIG.setString("state.checkpoints.dir", stateBackend);
> savepoint will be transfered from task manager to job manager (same as
> using
> memory as state backend), and stored in the disk of job manager?
> Savepoint seems to be necessary to be stored in the place where job manager
> can find, and then it can start the job again. It looks like it can work
> without s3 or hdfs , is that true? ( if this works, that means we can use
> EBS)
> Best,
> Sendoh
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/quesiont-
> related-to-store-location-of-save-point-when-using-file-
> instead-of-s3-and-hdfs-tp12435.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.

View raw message