flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ovidiu Cristian Marcu <ovidiu.cristian.ma...@huawei.com>
Subject RE: [FLINK-3035] Redis as State Backend
Date Mon, 07 Nov 2016 11:56:19 GMT
Thanks, how big was your state (GBs)?
Can you share your benchmark/s?


-----Original Message-----
From: amir bahmanyari [mailto:amirtousa@yahoo.com.INVALID] 
Sent: Tuesday, October 25, 2016 7:24 PM
To: dev@flink.apache.org
Subject: Re: [FLINK-3035] Redis as State Backend

FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck.
Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps
and it settled down tremendously.Amir-

      From: Ovidiu-Cristian MARCU <ovidiu-cristian.marcu@inria.fr>
 To: dev@flink.apache.org
 Sent: Tuesday, October 25, 2016 12:40 AM
 Subject: Re: [FLINK-3035] Redis as State Backend
Thank you!


> On 24 Oct 2016, at 16:11, Aljoscha Krettek <aljoscha@apache.org> wrote:
> Hi,
> regarding RocksDB, yes this is possible because RocksDB is essentially 
> only used as an out-of-core hash table. When checkpointing we write 
> everything from RocksDB to HDFS. When restoring we repopulate an empty 
> local RocksDB instance from the data in HDFS.
> Cheers,
> Aljoscha
> On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < 
> ovidiu.cristian.marcu@huawei.com> wrote:
>> Hi
>> I missed your reply, thank you for feedback.
>> Agree with 1, that will be possible only with Ramcloud.
>> It is clear the second point.
>> A short question: if you checkpoint the operator's state in hdfs I 
>> assume that on failure you are restarting the operator's tasks on 
>> other nodes, is that possible with RocksDB?
>> Best,
>> Ovidiu
>> -----Original Message-----
>> From: Aljoscha Krettek [mailto:aljoscha@apache.org]
>> Sent: Monday, October 17, 2016 2:51 PM
>> To: dev@flink.apache.org
>> Subject: Re: [FLINK-3035] Redis as State Backend
>> Hi,
>> there are two basic ideas for implementing a StateBackend based on Redis:
>> 1. Rely on Redis to keep the state, use nothing else.
>> 2. Use Redis to keep the state and checkpoint to some distributed 
>> file system (such as HDFS) when checkpointing
>> The first idea seems unwise because Redis is not a "strongly 
>> consistent distributed data store" as Elias pointed out on the issue. 
>> The second Idea is problematic because there is no easy way to read 
>> all state for a given Flink operator from a running Redis instance to 
>> store it in HDFS. That's what I was getting at in my comment.
>> Cheers,
>> Aljoscha
>> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < 
>> ovidiu.cristian.marcu@huawei.com> wrote:
>>> Hi
>>> Can you please expand the last comment:
>>> "I think, however, that for other reasons we will probably not be 
>>> able to implement this well. The problem is that we have to somehow 
>>> get at the state in redis for checkpointing. And if we use only one 
>>> Redis instance for all states then this will be problematic." - 
>>> Aljoscha Krettek
>>> Any other update on this issue will help, not clear the status.
>>> Best,
>>> Ovidiu

View raw message