flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aljoscha <...@git.apache.org>
Subject [GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Date Wed, 10 Aug 2016 09:27:51 GMT
Github user aljoscha commented on the issue:

    @wenlong88 I was talking about this effort to enable key-group sharding in Flink: https://issues.apache.org/jira/browse/FLINK-3755
    With this is becomes necessary to checkpoint the keyed states in a way that operators
can restore the state for the key shard for which they are responsible. The state backend
therefore has to write the snapshot data such that keys for different key-groups are separated.
If we keep the approach of simply checkpointing the whole RocksDB DB we don't get this property.
In the fully-async checkpointing mode we iterate over the keys for each key group and write
them separately from the keys of the other key groups. (logically speaking. they can still
go into the same file but we need an index that tells us where the state for each key group

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message