flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Yao <g...@apache.org>
Subject Re: Two questions about Async
Date Wed, 22 Apr 2020 16:22:50 GMT
> Bytes Sent but Records Sent is always 0

Sounds like a bug. However, I am unable to reproduce this using the
AsyncIOExample [1]. Can you provide a minimal working example?


> Is there an Async Sink? Or do I just rewrite my Sink as an AsyncFunction
followed by a dummy sink?

You will have to implement your own AsyncFunction to use as a sink. However, the
AsyncFunction operator does not need to be followed by a dummy sink.


> What’s the recommendation if the sink performing blocking I/O is proven to be
the root cause of back pressure?

There are many things that can be done, such as increasing the parallelism or
reducing per-record fixed costs. Note that all sinks are typically dominated by
I/O costs. It is difficult to give a recommendation without knowing details
about your use case and desired consistency guarantees.

Best,
Gary

[1] https://github.com/apache/flink/blob/aa4eb8f0c9ce74e6b92c3d9be5dc8e8cb536239d/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java


On Tue, Apr 21, 2020 at 11:06 PM Stephen Connolly
<stephen.alan.connolly@gmail.com> wrote:
>
> 1. On Flink 1.10 when I look at the topology overview, the AsyncFunctions show non-zero
values for Bytes Received; Records Received; Bytes Sent but Records Sent is always 0... yet
the next step in the topology shows approx the same Bytes Received as the async sent (modulo
minor delays) and a non-zero Records Received. Is the “Records Sent of an AsyncFunction
is always displayed as zero” a bug?
>
> 2. Is there an Async Sink? Or do I just rewrite my Sink as an AsyncFunction followed
by a dummy sink? What’s the recommendation if the sink performing blocking I/O is proven
to be the root cause of back pressure?
>
> Thanks in advance for your help
> --
> Sent from my phone

Mime
View raw message