flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: s3 statebackend user state size
Date Tue, 10 May 2016 17:33:16 GMT
On Tue, May 10, 2016 at 5:07 PM, Chen Qin <qinnchen@gmail.com> wrote:
> Future, to keep large key/value space, wiki point out using rocksdb as
> backend. My understanding is using rocksdb will write to local file systems
> instead of sync to s3. Does flink support memory->rocksdb(local disk)->s3
> checkpoint state split yet? Or would implement kvstate interface makes flink
> take care of large state problem?

Hey Chen,

when you use RocksDB, you only need to explicitly configure the file
system checkpoint directory, for which you can use S3:

new RocksDBStateBackend(new URI("s3://..."))

The local disk path are configured via the general Flink temp
directory configuration (see taskmanager.tmp.dirs in
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html,
default is /tmp).

State is written to the local RocksDB instance and the RocksDB files
are copied to S3 on checkpoints.

Does this help?

– Ufuk

Mime
View raw message