flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaogang Shi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5036) Perform the grouping of keys in restoring instead of checkpointing
Date Wed, 09 Nov 2016 03:27:58 GMT
Xiaogang Shi created FLINK-5036:
-----------------------------------

             Summary: Perform the grouping of keys in restoring instead of checkpointing
                 Key: FLINK-5036
                 URL: https://issues.apache.org/jira/browse/FLINK-5036
             Project: Flink
          Issue Type: Bug
          Components: State Backends, Checkpointing
            Reporter: Xiaogang Shi


Whenever taking snapshots of {{RocksDBKeyedStateBackend}}, the values in the states will be
written onto different files according to their key groups. The procedure is very costly when
the states are very big. 

Given that the snapshot operations will be performed much more frequently than restoring,
we can leave the key groups as they are to improve the overall performance. In other words,
we can perform the grouping of keys in restoring instead of in checkpointing.

I think, the implementation will be very similar to the restoring of non-partitioned states.
Each task will receive a collection of snapshots each of which contains a set of key groups.
Each task will restore its states from the given snapshots by picking values in assigned key
groups.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message