flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishnu Viswanath <vishnu.viswanat...@gmail.com>
Subject Re: Incremental checkpoint branch
Date Mon, 06 Mar 2017 16:38:20 GMT
Thanks everyone for the reply.

@Xiaogang design document looks very good, looking forward to check the
implementation and to try it out. I see that hdfs:// is mentioned as the
backend storage in the documentation, what about s3://. Is it something
that needed to be considered in the design, or is it handled by RocksDB.

Thanks,
Vishnu

On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <shixiaogangg@gmail.com> wrote:

> Hi Vinshnu,
>
> We have obtained an initial design of incremental checkpointing [1] and
> will start working on incremental checkpointing the next week. You can
> watch the issue  FLINK-5053 [2] to get timely notification of the updates.
> All suggestions are welcome.
>
> [1]
> https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS
> UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z
> [2] https://issues.apache.org/jira/browse/FLINK-5053
>
> Regards,
> Xiaogang
>
>
>
> 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <wshaoxuan@gmail.com>:
>
> > Vinshnu,
> > You can find the latest design discussion for incremental checkpoint in
> > http://apache-flink-mailing-list-archive.1008284.n3.
> > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-
> Flink-td15931.html
> >  @Stefan Richter
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter>
> > and @Xiaogang
> > Shi
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> name=xiaogang.shi>
> > might
> > be the right person to provide you its on-going status.
> >
> > cheers,
> > Shaoxuan
> >
> > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath <
> > vishnu.viswanath25@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Can someone point me to the branch where the ongoing work for
> incremental
> > > checkpoint is going on, I would like to try it out even if the work is
> > not
> > > complete.
> > >
> > > I have a use case where the state size increase about ~1gb every 5
> > minutes.
> > >
> > > Thanks,
> > > Vishnu
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message