Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 26C1A200B56 for ; Sat, 30 Jul 2016 20:46:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1924B160A8A; Sat, 30 Jul 2016 18:46:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5E8D2160A63 for ; Sat, 30 Jul 2016 20:46:20 +0200 (CEST) Received: (qmail 70741 invoked by uid 500); 30 Jul 2016 18:46:19 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 70731 invoked by uid 99); 30 Jul 2016 18:46:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Jul 2016 18:46:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 01EF8C26D9 for ; Sat, 30 Jul 2016 18:46:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.108 X-Spam-Level: X-Spam-Status: No, score=-0.108 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.287, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id GWBfduIC1h2I for ; Sat, 30 Jul 2016 18:46:14 +0000 (UTC) Received: from nm16-vm6.bullet.mail.ne1.yahoo.com (nm16-vm6.bullet.mail.ne1.yahoo.com [98.138.91.109]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id ECE095F47A for ; Sat, 30 Jul 2016 18:46:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1469904367; bh=QCwoE9G96JrFDgEDp5i6eTjFbXYFZqqz2Mer2d3h0G4=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=c3Gly8VAm5QXJYsaeIiMcpunDsgJPl2ahG252F1WkbsFnAf8g0JrjtS2PTj5SXW7e0fXRFwh6QEUtO+tek+lJJPbuqCdBlXAljHY5xsd//DFOD7heQrlse0f1K7reqql4U5CMokhqwHpyVVy8w9JxmUeesxWhLxq6esqS3bIB8k7UPE4UFFuBfS1qE56iSY3fFGnVHzIoQkTLrxqfFLq5lWUZz5iYx35X1D8ArLX66ASdsNqieLMixuSBY9LUzTeJaHvcNrFVmXYdfxEAkvw/x71YlF+BiA8kpNLsidd0PSO/REGA9G1YJOmSynQ+k/Sb1rg5tDRXtieuK7YF0WrnA== Received: from [98.138.101.130] by nm16.bullet.mail.ne1.yahoo.com with NNFMP; 30 Jul 2016 18:46:07 -0000 Received: from [98.138.226.161] by tm18.bullet.mail.ne1.yahoo.com with NNFMP; 30 Jul 2016 18:46:07 -0000 Received: from [127.0.0.1] by omp1062.mail.ne1.yahoo.com with NNFMP; 30 Jul 2016 18:46:07 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 585111.11221.bm@omp1062.mail.ne1.yahoo.com X-YMail-OSG: fRa67U0VM1kDDY3Wp7dIGCo3GMa4lvTRlKEWzhp.xIlrIKowyKsYIT4Xl_f5MEk dIzT3Ge6MoSKN68c6fsvyZuENsh6Y1.fsJbEwwTLUNhInJZ9ihjMSh4KnJaqs0yL8zbwSXiu87.p fyQ5MU5Ud7cwuLd06ou_1i5dXLkVupmv_176dmTnHaif0j9YU6kuy7KQhcPWzj9fK.7EWbBeQWIA ZOUWF_pGOfF1BHyC5XKuyCZRq4tT.oPqln5x5EhAOtXbFuwHxGTq.rbdZSuUtj5_VQbW7.KYhy.t Il9t4psLCluESodnkdvp77z_JnyfpnTvo3t4v8vkR0ha0NcNpaEYY8.ZXmK0oEK5wdoZrf5A_2Gm rZKu9GGfWXDsYn2wR7PmAMl8IN9Zc1NrAAE_y1uoXUjrzPSo5wcCSaL8MRED44y_eaGmIN1g0QZr ngn0dXUPQ__evz4PndYeJMJlMliX.DKUijuLGtkkRMtdnNbPPZG_tkC.4ENRe96yLidjrxCWwPtL 2cUP.MR3yC0LOPe.kvNzZNc3dpqx3_nJpUKgeFOTOUKb6oEc3M2jQapNt Received: from jws10013.mail.ne1.yahoo.com by sendmailws151.mail.ne1.yahoo.com; Sat, 30 Jul 2016 18:46:07 +0000; 1469904367.203 Date: Sat, 30 Jul 2016 18:46:06 +0000 (UTC) From: M Singh Reply-To: M Singh To: "user@flink.apache.org" Message-ID: <2041251988.6696293.1469904366424.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: <157237169.6364488.1469813162673.JavaMail.yahoo.ref@mail.yahoo.com> <157237169.6364488.1469813162673.JavaMail.yahoo@mail.yahoo.com> Subject: Re: Flink - Once and once processing MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6696292_1306704638.1469904366421" archived-at: Sat, 30 Jul 2016 18:46:21 -0000 ------=_Part_6696292_1306704638.1469904366421 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Konstantin. Just to clarify - unless the target database is resilient to duplicates, Fl= ink's once-only configuration will not avoid duplicate updates. Mans=20 On Saturday, July 30, 2016 7:40 AM, Konstantin Knauf wrote: =20 Hi Mans, depending on the number of operations and the particular database, you might be able to use transactions. Maybe you can also find a data model, which is more resilient to these kind of failures. Cheers, Konstantin On 29.07.2016 19:26, M Singh wrote: > Hi: >=20 > I have a use case where we need to update a counter in a db and for this > need to guarantee once only processing.=C2=A0 If we have some entries in = a > batch and it partially updates the counters and then fails, if Flink > retries the processing for that batch, some of the counters will be > updated twice (the ones which succeeded in the first batch). >=20 > I think in order to guarantee once only processing, I will have to set > the buffer size to zero (ie, send one item at a time). >=20 > Is there any alternative configuration or suggestion on how I can > achieve once only updates using a batch mode with partial failures ? >=20 > Thanks >=20 > Mans >=20 --=20 Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert Dahl= ke Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082 ------=_Part_6696292_1306704638.1469904366421 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Konstantin.

Just to clarify = - unless the target database is resilient to duplicates, Flink's once-only = configuration will not avoid duplicate updates.

<= span id=3D"yui_3_16_0_ym19_1_1469904223706_3205">Mans


On Saturday, July 30, 2016 7:40 AM, Konstantin Knauf <konstanti= n.knauf@tngtech.com> wrote:


Hi Mans,

depending on th= e number of operations and the particular database, you
m= ight be able to use transactions.

Mayb= e you can also find a data model, which is more resilient to these
kind of failures.

Cheers,
Konstantin

On 29.07.2016 19:26= , M Singh wrote:
> Hi:
>
> I have a use case where we need to update a counter in a db= and for this
> need to guarantee once only processing= .  If we have some entries in a
> batch and it pa= rtially updates the counters and then fails, if Flink
>= ; retries the processing for that batch, some of the counters will be
> updated twice (the ones which succeeded in the first batc= h).
>
> I think in order to guar= antee once only processing, I will have to set
> the b= uffer size to zero (ie, send one item at a time).
> > Is there any alternative configuration or suggestion = on how I can
> achieve once only updates using a batch= mode with partial failures ?
>
>= ; Thanks
>
> Mans

>

--
= Konstantin Knauf * konstantin.knauf@tng= tech.com * +49-174-3413182
TNG Technology Consulting = GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring
Gesch=C3=A4ft= sf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082

<= br>
------=_Part_6696292_1306704638.1469904366421--