flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ovidiu-Cristian MARCU <ovidiu-cristian.ma...@inria.fr>
Subject Re: [FLINK-3035] Redis as State Backend
Date Tue, 25 Oct 2016 07:40:59 GMT
Thank you!

Best,
Ovidiu

> On 24 Oct 2016, at 16:11, Aljoscha Krettek <aljoscha@apache.org> wrote:
> 
> Hi,
> regarding RocksDB, yes this is possible because RocksDB is essentially only
> used as an out-of-core hash table. When checkpointing we write everything
> from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB
> instance from the data in HDFS.
> 
> Cheers,
> Aljoscha
> 
> On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu <
> ovidiu.cristian.marcu@huawei.com> wrote:
> 
>> Hi
>> 
>> I missed your reply, thank you for feedback.
>> Agree with 1, that will be possible only with Ramcloud.
>> It is clear the second point.
>> 
>> A short question: if you checkpoint the operator's state in hdfs I assume
>> that on failure you are
>> restarting the operator's tasks on other nodes, is that possible with
>> RocksDB?
>> 
>> Best,
>> Ovidiu
>> 
>> -----Original Message-----
>> From: Aljoscha Krettek [mailto:aljoscha@apache.org]
>> Sent: Monday, October 17, 2016 2:51 PM
>> To: dev@flink.apache.org
>> Subject: Re: [FLINK-3035] Redis as State Backend
>> 
>> Hi,
>> there are two basic ideas for implementing a StateBackend based on Redis:
>> 1. Rely on Redis to keep the state, use nothing else.
>> 2. Use Redis to keep the state and checkpoint to some distributed file
>> system (such as HDFS) when checkpointing
>> 
>> The first idea seems unwise because Redis is not a "strongly consistent
>> distributed data store" as Elias pointed out on the issue. The second Idea
>> is problematic because there is no easy way to read all state for a given
>> Flink operator from a running Redis instance to store it in HDFS. That's
>> what I was getting at in my comment.
>> 
>> Cheers,
>> Aljoscha
>> 
>> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu <
>> ovidiu.cristian.marcu@huawei.com> wrote:
>> 
>>> Hi
>>> 
>>> Can you please expand the last comment:
>>> 
>>> "I think, however, that for other reasons we will probably not be able
>>> to implement this well. The problem is that we have to somehow get at
>>> the state in redis for checkpointing. And if we use only one Redis
>>> instance for all states then this will be problematic." - Aljoscha
>>> Krettek
>>> 
>>> Any other update on this issue will help, not clear the status.
>>> 
>>> Best,
>>> Ovidiu
>>> 
>>> 
>> 


Mime
View raw message