apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <thomas.we...@gmail.com>
Subject Re: Datatorrent fault tolerance
Date Thu, 06 Oct 2016 22:47:01 GMT
In that case please have a look at:

https://github.com/apache/apex-malhar/blob/master/kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java

The operator will ensure that messages are not duplicated, under the stated
assumptions.


On Thu, Oct 6, 2016 at 3:37 PM, Jaspal Singh <jaspal.singh1404@gmail.com>
wrote:

> Hi Thomas,
>
> In our case we are writing the results back to maprstreams topic based on
> some validations.
>
>
> Thanks
> Jaspal
>
>
> On Thursday, October 6, 2016, Thomas Weise <thw@apache.org> wrote:
>
>> Hi,
>>
>> which operators in your application are writing to external systems?
>>
>> When you look at the example from the blog (https://github.com/DataTorren
>> t/examples/tree/master/tutorials/exactly-once), there is Kafka input,
>> which is configured to be idempotent. The results are written to JDBC. That
>> operator by itself supports exactly-once through transactions (in
>> conjunction with idempotent input), hence there is no need to configure the
>> processing mode at all.
>>
>> Thomas
>>
>>

Mime
View raw message