flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode
Date Sat, 13 Aug 2016 15:57:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419978#comment-15419978

ASF GitHub Bot commented on FLINK-4340:

Github user wenlong88 commented on the issue:

    @StephanEwen you are right. But in specific situation, we may need some temporary compromise
do make the system work well, and then remove the  compromised points latter as soon as possible
    I think both approaches have shortcomes.  When the state is large such as millions of
KVs per db, full async approach can do the full async backup, but will cost a lot of time
to restore which may be intolerable while doing fail-over in production. So I think it is
necessary to have both, and the full async can be the default option. 
    Considering that there is no really perfect solution yet, I think It is OK to remove the
semi-async way right now to avoid blocking the job of key group but need to reintroduce a
better solution latter soon if you agree that rocksdb is quite a good choice of statebackend
in large state situations. 
    Regrading to the overhead of memory in different dbs. Rocksdb can share the same block
cache for different db instance but I don't know how to reduce the cost of memtables which
is also a problem existed in current solution that allowing to store different stats in a
single db using column families since memtables of column families are also separated.

> Remove RocksDB Semi-Async Checkpoint Mode
> -----------------------------------------
>                 Key: FLINK-4340
>                 URL: https://issues.apache.org/jira/browse/FLINK-4340
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.1.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
> This seems to be causing to many problems and is also incompatible with the upcoming
key-group/sharding changes that will allow rescaling of keyed state.
> Once this is done we can also close FLINK-4228.

This message was sent by Atlassian JIRA

View raw message