apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hegde <sand...@datatorrent.com>
Subject Re: At-least once semantics for Kafka to Cassndra ingest
Date Tue, 14 Feb 2017 22:36:18 GMT
Settings mentioned by Sanjay, will only guarantee exactly once for Windows,
but not for partial window processed by the operator, in a way that setting
is a misnomer.
To achieve Exactly once, there are some precoditions that need to be met
along with the support in the output operator. Here is a blog that gives
the idea about exactly once,
https://www.datatorrent.com/blog/end-to-end-exactly-once-with-apache-apex/

On Tue, Feb 14, 2017 at 2:11 PM Sanjay Pujare <sanjay@datatorrent.com>
wrote:

> Have you taken a look at
> http://apex.apache.org/docs/apex/application_development/#exactly-once ?
> i.e. setting that processing mode on all the operators in the pipeline .
>
> Join us at Apex Big Data World-San Jose <
> http://www.apexbigdata.com/san-jose.html>, April 4, 2017!
>
> http://www.apexbigdata.com/san-jose-register.html
>
>
> On 2/14/17, 12:00 PM, "Himanshu Bari" <himanshubari@gmail.com> wrote:
>
>     How to ensure that the Kafka to Cassandra ingestion pipeline in Apex
> will
>     guarantee exactly once processing semantics.
>     Eg. Message was read from Kafka but apex app died before it was written
>     successfully to Cassandra.
>
>
>
> --
*Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
[image: http://www.apexbigdata.com/san-jose-register.html]
<http://www.apexbigdata.com/san-jose-register.html>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message