flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From M Singh <mans2si...@yahoo.com>
Subject Re: Flink - Once and once processing
Date Sat, 30 Jul 2016 18:46:06 GMT
Thanks Konstantin.
Just to clarify - unless the target database is resilient to duplicates, Flink's once-only
configuration will not avoid duplicate updates.

    On Saturday, July 30, 2016 7:40 AM, Konstantin Knauf <konstantin.knauf@tngtech.com>

 Hi Mans,

depending on the number of operations and the particular database, you
might be able to use transactions.

Maybe you can also find a data model, which is more resilient to these
kind of failures.



On 29.07.2016 19:26, M Singh wrote:
> Hi:
> I have a use case where we need to update a counter in a db and for this
> need to guarantee once only processing.  If we have some entries in a
> batch and it partially updates the counters and then fails, if Flink
> retries the processing for that batch, some of the counters will be
> updated twice (the ones which succeeded in the first batch).
> I think in order to guarantee once only processing, I will have to set
> the buffer size to zero (ie, send one item at a time).
> Is there any alternative configuration or suggestion on how I can
> achieve once only updates using a batch mode with partial failures ?
> Thanks
> Mans

Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

View raw message