flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: 答复:how to understand the flink flow control
Date Mon, 10 Aug 2015 09:12:52 GMT
Good to hear. Answers to your questions are inline.

I think we should move the discussion to the dev@flink.a.o list though if
there are more questions.

On Mon, Aug 10, 2015 at 9:29 AM, wangzhijiang999 <wangzhijiang999@aliyun.com
> wrote:

> Hi Ufuk,
>
>
>        Thank you for your detail and clear explaination!  I reviewed the
> code based on your info and got it clearly.
>
>
> For local transfer:
>
>    1.
>
>    When producer emits results to ResultPartition, it will request
>     Buffer from the pool. If there  are no available Buffer, it will wait .
>
>
Yes, the buffer request is blocking.


>
>    1.
>
>    The ResultPartition of producer is the InputGate of Consumer, when
>    consumer read buffer from InputGate and deserializer the buffer, the Buffer
>    will be recycled ,so the producer can request Buffer again.
>
>
The producer never requests a buffer from the input gate, but from its own
pool. If the input gate is slow to recycle its buffers, the producer will
become slow as well. But you are right that the producer will not be able
to request a buffer if the input gate never recycles them.


>
>    1.
>
>    So if the consumer slows, the Buffer in the ResultPartition of
>    producer can not be recycled quickly, resulting in producer has no
>    available Buffer to emit data.
>    2.
>
>    If the producer waits the available Buffer to emit, it can not process
>    elements and read next data from its InputGate, resulting in slowing the
>    producer.
>
> Yes. The task thread is both consuming and producing data. So if its
blocked on the output side, the input is not consumed, which propagates
back pressure further.


> For remote transfer:
>
>      The InputGate and ResultPartition for each task are separate
>
>      1. For producer the Buffer in ResultPartition will be recycled when
> producer write them to the channel.
>
>      2. For consumer the Buffer will be recycled when consumer read it
> from InputGate and deserializer it . The consumer need to request Buffer
> when reading data from channel     and put them to InputGate.
>
>      3.  When consumer slows and there are no available Buffer , the
> consumer will not read data from channel, so it will affect the producer
> writing data to channel and  the producer will have no available Buffer to
> emit result at last resulting in slowing producer.
>

Yes but there is some logic involved to stop reading from the actual TCP
channel to back pressure the remote producer. Essentially the selector read
event is unsubscribed and on the output side there is Netty's watermark
mechanism, which prevents further data from being written to the TCP
channel.

Mime
View raw message