flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sihua Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8845) Introduce `parallel recovery` mode for full checkpoint (savepoint)
Date Wed, 07 Mar 2018 06:43:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389123#comment-16389123

Sihua Zhou commented on FLINK-8845:

Unfortunately, even though according to RocksDB [wiki|https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ],
the best way to load data into RocksDB is "Generate SST files (using {{SstFileWriter}}) with
non-overlapping ranges in parallel and bulk load the SST files.". But after implementing this
and test with a simple bench mark, I found that the performance is not that good as expected,
it's almost the same or worst that as using {{Rocks.put()}}. After a bit analysis I found
that when building SST it consumed a lot of time to create {{DirectSlice}} and currently we
can't reuse the {{DirectSlice}} in java api. Even though in C++ this could help to get a outperformance
result, but in java I think we can't use this to improve the performance currently (maybe
somedays RocksDB might improve this to enable us get a approximate performance in java as
using C++) ...

>  Introduce `parallel recovery` mode for full checkpoint (savepoint)
> -------------------------------------------------------------------
>                 Key: FLINK-8845
>                 URL: https://issues.apache.org/jira/browse/FLINK-8845
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.5.0
>            Reporter: Sihua Zhou
>            Assignee: Sihua Zhou
>            Priority: Major
>             Fix For: 1.6.0
> Base on {{ingestExternalFile()}} and {{SstFileWriter}} provided by RocksDB, we can restore
from fully checkpoint (savepoint) in parallel. This can also be extended to incremental checkpoint
easily, but for the sake of simple, we do this in two separate tasks.

This message was sent by Atlassian JIRA

View raw message