flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Nowojski <pi...@ververica.com>
Subject Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter
Date Thu, 09 Apr 2020 08:20:29 GMT
Hi,

With:

> Can not you take into account the pending element that’s stuck somewhere in the transit?
Snapshot it as well and during recovery reprocess it? This is exactly that’s AsyncWaitOperator
is doing.

I didn’t mean for you to use AsynWaitOperator, but what both me and Arvid suggested you
previously:

> Also as Kostas pointed out, the easiest way would be to try use AsyncWaitOperator. If
that’s not possible, you can implement your custom logic based on its code.

You can copy/duplicate & modify/adjust the AsyncWaitOperator logic inside your custom
operator. You don’t have to use it if you have some special requirements, you can implement
your own custom logic. Specifically I meant to mimic 

org.apache.flink.streaming.api.operators.async.AsyncWaitOperator#queue

Field and how is it being used during snapshotting state & recovery.

Piotrek

> On 9 Apr 2020, at 06:10, Salva Alcántara <salcantaraphd@gmail.com> wrote:
> 
> I agree with your point Piotrek, AsyncIO would handle all the pending data
> for me. However, the reason why I did not want to use it is because in my
> case, the callbacks are not always called in response of new data being sent
> to the third party lib. Indeed, the callback will be called rather
> uncommonly (since in my case it will mean that an anomaly has been
> detected). This means that If I go with AsyncIO I will need to setup a max
> timeout for every element, when only a few of them will actuallyinvoke the
> callback (i.e., produce any data in response). This seems rather drastic
> because it will probably add too much latency unnecessarily, but I agree on
> that maybe there is no other way if I need exactly once guarantees.
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/


Mime
View raw message