flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sjk <shijinkui...@163.com>
Subject Re: [Discuss] State Backend use external HBase storage
Date Thu, 17 Nov 2016 10:23:18 GMT
Hi, Chen Qin
I fount this issue. Does it kicked off?  What’s the current progress?
https://issues.apache.org/jira/browse/FLINK-4266 <https://issues.apache.org/jira/browse/FLINK-4266>

> On Nov 16, 2016, at 19:35, Till Rohrmann <trohrmann@apache.org> wrote:
> Hi Jinkui,
> the file system state backend and the RocksDB state backend can be
> configured (and usually should be) such that they store their checkpoint
> data on a reliable storage system such as HDFS. Then you also have the
> reliability guarantees.
> Of course, one can start adding more state backends to Flink. At some point
> in time there was the idea to write a Cassandra backed state backend [1],
> for example. Similarly, one could think about a HBase backed state backend.
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Cassandra-statebackend-td12690.html
> Cheers,
> Till
> On Wed, Nov 16, 2016 at 3:10 AM, shijinkui <shijinkui@huawei.com> wrote:
>> Hi, All
>> At present flink have three state backend: memory, file system, rocksdb.
>> MemoryStateBackend will tansform the snapshot to jobManager, 5MB limited
>> default. Even setting it bigger, that not suitable for very big state
>> storage.
>> HDFS can meet the reliability guarantee, but It's slow. File System and
>> RocksDB are fast, but they are have no reliability guarantee.
>> Three state backend all have no reliability guarantee.
>> Can we have a Hbase state backend, providing reliability guarantee of
>> state snapshot?
>> For user, only new a HbaseStateBackend object, provide hbase parameter and
>> optimization configure.
>> Maybe Hbase or other distributed key-value storage is heavyweight storage,
>> we only use hbase client to read/write asynchronously.
>> -Jinkui Shi

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message