flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangzhijiang999" <wangzhijiang...@aliyun.com>
Subject 回复:problem of sharing TCP connection when transferring data
Date Mon, 23 May 2016 16:55:34 GMT
 Hi Ufuk,
       Thank you for the detail explaination!  As we confirmed that the task will set
the autoread as false for the sharing channel when no available segment buffer. In further,
when this task has available buffer again, it will notify the event to set the autoread as
true. But in some scenarios, there would be a propobility that the autoread for this sharing
channel would not be set as true anymore. That is , when available buffer to notify event
and currently there are some messages staged in the queue,it would process these messages
first, the message shoule be put on input channel buffer in common way, but if the task failed
and the buffer pool is released, it will return false when process the message,so the channel
will not be set as autoread true any more, then all the other tasks sharing this channel will
be effected.      In summary, if one task set autoread as false, and when it notify the
available buffer, there are some messages during this time to be processed first, if one message
belongs to another failed task, the autoread for this channel would not be set true anymore.
The only way is to cancel all the tasks in this channel to release the channel. Is it right?   
In the past, I improved the failover strategy based on flink for our application and noticed
this issue. Also i am very interested and pleasure to do some related work for flink improvement
as you mentioned. Actually i am working on improving flink in many ways for our application,
and wish further contact with you for the professional advise. Thank you again!
 Zhijiang Wang------------------------------------------------------------------发件人:Ufuk
Celebi <uce@apache.org>发送时间:2016年5月23日(星期一) 19:49收件人:user
<user@flink.apache.org>; wangzhijiang999 <wangzhijiang999@aliyun.com>主 题:Re:
problem of sharing TCP connection when transferring data Yes, that is a correct description of the state of things.

A way to improve this is to introduce flow control in the application
layer, where consumers only receive buffers when they have buffers
available. They could announce on the channel how many buffers they
have before they receive anything. This way there will be no blocking
of the channel and we could actually multiplex more consumers over the
same channel.

The implementation is probably a little tricky, but if you want to
work on this and have time to actually do it, we can think about the
details. :-) Would you be interested? If yes, let's schedule a Hangout
where we brainstorm about the solution and how to implement it.
Ideally, we would come up with a design document, which we share on
the mailing list and then we continue implementing it. I currently
only have time to act as a guide/mentor and you would have to do most
of the implementation.

– Ufuk

On Mon, May 23, 2016 at 5:40 AM, wangzhijiang999
<wangzhijiang999@aliyun.com> wrote:
> Hi,
>      I am confused with sharing tcp connection for the same connectionID, if
> two tasks share the same connection, and there is no available buffer in the
> local buffer pool of the first task  , then it will set autoread as false
> for the channel, but it will effect the second task if it still has
> available buffer. So if one of the tasks no available buffer , all the other
> tasks can not read data from channel because of this. My understanding is
> right? If so, are there any improvements for it?  Thank you for any help!
View raw message