Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EAF2C200A5B for ; Wed, 25 May 2016 14:24:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id EA36B160A18; Wed, 25 May 2016 12:24:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0B961160A29 for ; Wed, 25 May 2016 14:24:32 +0200 (CEST) Received: (qmail 90994 invoked by uid 500); 25 May 2016 12:15:12 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 90985 invoked by uid 99); 25 May 2016 12:15:12 -0000 Received: from Unknown (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2016 12:15:12 +0000 Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 224EA1A0098 for ; Wed, 25 May 2016 12:14:50 +0000 (UTC) Received: by mail-lf0-f48.google.com with SMTP id e130so17349225lfe.3 for ; Wed, 25 May 2016 05:14:49 -0700 (PDT) X-Gm-Message-State: ALyK8tIAer2SeitpJOPdDSNzduwpF/XOajiwOEn5qcJFB82LR6te6t2JKSd05Wh/NuziVR7je+MRLP9gLeLxvg== X-Received: by 10.25.22.165 with SMTP id 37mr893285lfw.100.1464178488022; Wed, 25 May 2016 05:14:48 -0700 (PDT) MIME-Version: 1.0 References: <20882204.1634631464161178776.JavaMail.root@correo> <9408737.1634771464161339122.JavaMail.root@correo> In-Reply-To: From: Aljoscha Krettek Date: Wed, 25 May 2016 12:14:38 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Incremental updates To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a113f239eaaa2570533a99e8c archived-at: Wed, 25 May 2016 12:24:34 -0000 --001a113f239eaaa2570533a99e8c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, right now, this does not work but we're is also actively working on that. This is the design doc for part one of the necessary changes: https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9Wyss= Q63w/edit?usp=3Dsharing Cheers, Aljoscha On Wed, 25 May 2016 at 13:32 Malgorzata Kudelska wrote: > Hi, > Thanks for your reply. > > Is Flink able to detect that an additional server joined and rebalance th= e > processing? How is it done if I have a keyed stream and some custom > ValueState variables? > > Cheers, > Gosia > > 2016-05-25 11:32 GMT+02:00 Aljoscha Krettek : > >> Hi Gosia, >> right now, Flink is not doing incremental checkpoints. Every checkpoint >> is fully valid in isolation. Incremental checkpointing came up several >> times on ML discussions and we a planning to work on it once someone fin= ds >> some free time. >> >> Cheers, >> Aljoscha >> >> On Wed, 25 May 2016 at 09:29 Rub=C3=A9n Casado >> wrote: >> >>> Hi Gosia >>> >>> You can have a look to the PROTEUS project we are doing [1]. We are >>> implementing incremental version of analytics operations. For example y= ou >>> can see in [2] the implementation of the incremental AVG. Maybe the cod= e >>> can give you some ideas :-) >>> >>> >>> [1] https://github.com/proteus-h2020/proteus-backend/tree/development >>> [2] >>> https://github.com/proteus-h2020/proteus-backend/blob/development/src/m= ain/java/com/treelogic/proteus/flink/incops/IncrementalAverage.java >>> >>> ______________________________________ >>> >>> *Dr. Rub=C3=A9n Casado* >>> Head of Big Data >>> Treelogic >>> >>> *ruben.casado.treelogic* >>> >>> +34 902 286 386 - +34 607 18 28 06 >>> Parque Tecnol=C3=B3gico de Asturias =C2=B7 Parcela 30 >>> E33428 Llanera =C2=B7 Asturias [Spain] >>> www.treelogic.com >>> ______________________________________ >>> >>> >>> ----- Mensaje original ----- >>> De: "Malgorzata Kudelska" >>> Para: user@flink.apache.org >>> Enviados: Martes, 24 de Mayo 2016 22:01:28 GMT +01:00 Amsterdam / Berl= =C3=ADn >>> / Berna / Roma / Estocolmo / Viena >>> Asunto: Incremental updates >>> >>> >>> Hi, >>> I have the following question. Does Flink support incremental updates? >>> >>> In particular, I have a custom StateValue object and during the >>> checkpoints I would like to save only the fields that changed since the >>> previous checkpoint. Is that possible? >>> >>> Regards, >>> Gosia >>> >> > --001a113f239eaaa2570533a99e8c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
right now, this does not work but we're is als= o actively working on that. This is the design doc for part one of the nece= ssary changes:=C2=A0https://docs.google= .com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit?usp=3Dsha= ring

Cheers,
Aljoscha

On Wed, 25 May 2016 at 13:32 Malgorz= ata Kudelska <m.j.kudelska@gma= il.com> wrote:
Hi,
Thanks for your reply.

Is Flink able t= o detect that an additional server joined and rebalance the processing? How= is it done if I have a keyed stream and some custom ValueState variables?<= /div>

Cheers,=C2=A0
Gosia

2016-05-25 11:32 GMT+02:00= Aljoscha Krettek <aljoscha@apache.org>:
Hi Gosia,
right now, Flink is not doi= ng incremental checkpoints. Every checkpoint is fully valid in isolation. I= ncremental checkpointing came up several times on ML discussions and we a p= lanning to work on it once someone finds some free time.

Cheers,
Aljoscha

On Wed, 25 May 2016 at 09:29 Rub=C3=A9n Casado = <ruben.c= asado@treelogic.com> wrote:
=
Hi Gosia=

You can have a look to the PROTEUS project we are doing [1]. We are= implementing incremental version of analytics operations. For example you = can see in [2] the implementation of the incremental AVG. Maybe the code ca= n give you some ideas :-)


[1] https://githu= b.com/proteus-h2020/proteus-backend/tree/development
[2] https://github.com/proteus-h2020/proteus-backend/blob/development/s= rc/main/java/com/treelogic/proteus/flink/incops/IncrementalAverage.java=

______________________________________

Dr. Rub=C3=A9n Casado
Head of Big Data
Treelogic
 ruben.casado.treelogic
+34 902 286 386 - +34 607 18 28 06=
Parque Tecnol=C3=B3gico de Asturias= =C2=B7 Parcela 30
E33428 Llanera = =C2=B7 Asturias [Spain]
www.= treelogic.com
______________________________________
=


----- M= ensaje original -----
De: "Malgorzata Kudelska" <m.j.kudelska@gmail.com>
Para: u= ser@flink.apache.org
Enviados: Martes, 24 de Mayo 2016 22:01:28 GMT = +01:00 Amsterdam / Berl=C3=ADn / Berna / Roma / Estocolmo / Viena
Asunto= : Incremental updates


Hi,
I have the following question. Does Flink support incremental updates?

In particular, I have a custom StateValue object and during = the checkpoints I would like to save only the fields that changed since the= previous checkpoint. Is that possible?

Regards,
Gosia


--001a113f239eaaa2570533a99e8c--