Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
References: <6ABEDF301876044D881B1E91C64368EAFA5D0B@lhreml502-mbs>
 <CANMXwW06xXzF+jU1SPce17tbnAxLCuPm+3z=XPtMdpNaL3TUFQ@mail.gmail.com> <6ABEDF301876044D881B1E91C64368EAFA6E70@lhreml502-mbs>
In-Reply-To: <6ABEDF301876044D881B1E91C64368EAFA6E70@lhreml502-mbs>
From: Aljoscha Krettek <aljoscha@apache.org>
Date: Mon, 24 Oct 2016 13:11:10 +0000
Message-ID: <CANMXwW3HjAe33CpBwErDPc=n_WVAza9G6Zqj8eMLGdjEh1kPOw@mail.gmail.com>
Subject: Re: [FLINK-3035] Redis as State Backend
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a1144955cc24033053f9c208b
archived-at: Mon, 24 Oct 2016 13:20:23 -0000

--001a1144955cc24033053f9c208b
Content-Type: text/plain; charset=UTF-8

Hi,
regarding RocksDB, yes this is possible because RocksDB is essentially only
used as an out-of-core hash table. When checkpointing we write everything
from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB
instance from the data in HDFS.

Cheers,
Aljoscha

On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu <
ovidiu.cristian.marcu@huawei.com> wrote:

> Hi
>
> I missed your reply, thank you for feedback.
> Agree with 1, that will be possible only with Ramcloud.
> It is clear the second point.
>
> A short question: if you checkpoint the operator's state in hdfs I assume
> that on failure you are
> restarting the operator's tasks on other nodes, is that possible with
> RocksDB?
>
> Best,
> Ovidiu
>
> -----Original Message-----
> From: Aljoscha Krettek [mailto:aljoscha@apache.org]
> Sent: Monday, October 17, 2016 2:51 PM
> To: dev@flink.apache.org
> Subject: Re: [FLINK-3035] Redis as State Backend
>
> Hi,
> there are two basic ideas for implementing a StateBackend based on Redis:
>  1. Rely on Redis to keep the state, use nothing else.
>  2. Use Redis to keep the state and checkpoint to some distributed file
> system (such as HDFS) when checkpointing
>
> The first idea seems unwise because Redis is not a "strongly consistent
> distributed data store" as Elias pointed out on the issue. The second Idea
> is problematic because there is no easy way to read all state for a given
> Flink operator from a running Redis instance to store it in HDFS. That's
> what I was getting at in my comment.
>
> Cheers,
> Aljoscha
>
> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu <
> ovidiu.cristian.marcu@huawei.com> wrote:
>
> > Hi
> >
> > Can you please expand the last comment:
> >
> > "I think, however, that for other reasons we will probably not be able
> > to implement this well. The problem is that we have to somehow get at
> > the state in redis for checkpointing. And if we use only one Redis
> > instance for all states then this will be problematic." - Aljoscha
> > Krettek
> >
> > Any other update on this issue will help, not clear the status.
> >
> > Best,
> > Ovidiu
> >
> >
>

--001a1144955cc24033053f9c208b--