Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 16CC7200A5B for ; Wed, 25 May 2016 11:32:53 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 15C33160A2E; Wed, 25 May 2016 09:32:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 32239160A18 for ; Wed, 25 May 2016 11:32:52 +0200 (CEST) Received: (qmail 17366 invoked by uid 500); 25 May 2016 09:32:51 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 17345 invoked by uid 99); 25 May 2016 09:32:51 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2016 09:32:51 +0000 Received: from mail-lf0-f53.google.com (mail-lf0-f53.google.com [209.85.215.53]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id ADD841A0098 for ; Wed, 25 May 2016 09:32:50 +0000 (UTC) Received: by mail-lf0-f53.google.com with SMTP id w16so3591258lfd.2 for ; Wed, 25 May 2016 02:32:50 -0700 (PDT) X-Gm-Message-State: ALyK8tI2lBeTMUkI4Hlt+XPjtV9uNhZv9CaFRPLnElyrNYJCoMWtZLb0OHYRNiGIFdq8Y0+z6irghBxJ6Y0SgQ== X-Received: by 10.25.27.135 with SMTP id b129mr728237lfb.102.1464168769222; Wed, 25 May 2016 02:32:49 -0700 (PDT) MIME-Version: 1.0 References: <20882204.1634631464161178776.JavaMail.root@correo> <9408737.1634771464161339122.JavaMail.root@correo> In-Reply-To: <9408737.1634771464161339122.JavaMail.root@correo> From: Aljoscha Krettek Date: Wed, 25 May 2016 09:32:39 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Incremental updates To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a114023126181530533a75b3c archived-at: Wed, 25 May 2016 09:32:53 -0000 --001a114023126181530533a75b3c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Gosia, right now, Flink is not doing incremental checkpoints. Every checkpoint is fully valid in isolation. Incremental checkpointing came up several times on ML discussions and we a planning to work on it once someone finds some free time. Cheers, Aljoscha On Wed, 25 May 2016 at 09:29 Rub=C3=A9n Casado wrote: > Hi Gosia > > You can have a look to the PROTEUS project we are doing [1]. We are > implementing incremental version of analytics operations. For example you > can see in [2] the implementation of the incremental AVG. Maybe the code > can give you some ideas :-) > > > [1] https://github.com/proteus-h2020/proteus-backend/tree/development > [2] > https://github.com/proteus-h2020/proteus-backend/blob/development/src/mai= n/java/com/treelogic/proteus/flink/incops/IncrementalAverage.java > > ______________________________________ > > *Dr. Rub=C3=A9n Casado* > Head of Big Data > Treelogic > > *ruben.casado.treelogic* > > +34 902 286 386 - +34 607 18 28 06 > Parque Tecnol=C3=B3gico de Asturias =C2=B7 Parcela 30 > E33428 Llanera =C2=B7 Asturias [Spain] > www.treelogic.com > ______________________________________ > > > ----- Mensaje original ----- > De: "Malgorzata Kudelska" > Para: user@flink.apache.org > Enviados: Martes, 24 de Mayo 2016 22:01:28 GMT +01:00 Amsterdam / Berl=C3= =ADn / > Berna / Roma / Estocolmo / Viena > Asunto: Incremental updates > > > Hi, > I have the following question. Does Flink support incremental updates? > > In particular, I have a custom StateValue object and during the > checkpoints I would like to save only the fields that changed since the > previous checkpoint. Is that possible? > > Regards, > Gosia > --001a114023126181530533a75b3c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Gosia,
right now, Flink is not doing incremental ch= eckpoints. Every checkpoint is fully valid in isolation. Incremental checkp= ointing came up several times on ML discussions and we a planning to work o= n it once someone finds some free time.

Cheers,
Aljoscha

On Wed, 25 May 2016 at 09:29 Rub=C3=A9n Casado <ruben.casado@treelogic.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Hi Gosia

You can have a look to the PROTEUS pr= oject we are doing [1]. We are implementing incremental version of analytic= s operations. For example you can see in [2] the implementation of the incr= emental AVG. Maybe the code can give you some ideas :-)


[1] https://github.com/proteus-h2020/proteus-backend/tree/deve= lopment
[2] https://github.com/proteus-h2020/prot= eus-backend/blob/development/src/main/java/com/treelogic/proteus/flink/inco= ps/IncrementalAverage.java

__________= ____________________________

<= font size=3D"2">Dr. Rub= =C3=A9n Casado
Head of Big Data<= br style=3D"font-family:arial,helvetica,sans-serif">Treelogic
ruben.= casado.treelogic

+34 902 286 386 - +34 607 18 = 28 06
Parque Tecnol=C3=B3gico de Ast= urias =C2=B7 Parcela 30
E33428 Llane= ra =C2=B7 Asturias [Spain]
w= ww.treelogic.com
______________________________________


----= - Mensaje original -----
De: "Malgorzata Kudelska" <m.j.kudelska@gmail.com= >
Para: user@flink.apache.org
Enviados: Martes, 24 de Mayo 2016 22:01:28 G= MT +01:00 Amsterdam / Berl=C3=ADn / Berna / Roma / Estocolmo / Viena
Asu= nto: Incremental updates


Hi,
I have the following question. Does Flink support incremental updates?

In particular, I have a custom StateValue object and during = the checkpoints I would like to save only the fields that changed since the= previous checkpoint. Is that possible?

Regards,
Gosia

--001a114023126181530533a75b3c--