flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7406) Implement Netty receiver incoming pipeline for credit-based
Date Tue, 21 Nov 2017 14:50:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260821#comment-16260821
] 

ASF GitHub Bot commented on FLINK-7406:
---------------------------------------

Github user NicoK commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4509#discussion_r152255026
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
---
    @@ -82,17 +84,19 @@
     	/** The initial number of exclusive buffers assigned to this channel. */
     	private int initialCredit;
     
    -	/** The current available buffers including both exclusive buffers and requested floating
buffers. */
    -	private final ArrayDeque<Buffer> availableBuffers = new ArrayDeque<>();
    +	/** The available buffer queue wraps both exclusive and requested floating buffers.
*/
    +	private final AvailableBufferQueue bufferQueue = new AvailableBufferQueue();
     
     	/** The number of available buffers that have not been announced to the producer yet.
*/
     	private final AtomicInteger unannouncedCredit = new AtomicInteger(0);
     
     	/** The number of unsent buffers in the producer's sub partition. */
    -	private final AtomicInteger senderBacklog = new AtomicInteger(0);
    +	@GuardedBy("bufferQueue")
    +	private int senderBacklog;
    --- End diff --
    
    Skipping over the code, we only ever need `numRequiredBuffers = senderBacklog + initialCredit`
- what do you think about storing this value instead of the backlog itself? It could still
be retrieved if desired, but we spare adding these two numbers over and over again and only
need to update this in `onSenderBacklog()`.


> Implement Netty receiver incoming pipeline for credit-based
> -----------------------------------------------------------
>
>                 Key: FLINK-7406
>                 URL: https://issues.apache.org/jira/browse/FLINK-7406
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>            Reporter: zhijiang
>            Assignee: zhijiang
>             Fix For: 1.5.0
>
>
> This is a part of work for credit-based network flow control.
> Currently {{PartitionRequestClientHandler}} receives and reads {{BufferResponse}} from
producer. It will request buffer from {{BufferPool}} for holding the message. If not got,
the message is staged temporarily and {{autoread}} for channel is set false.
> For credit-based mode, {{PartitionRequestClientHandler}} can always get buffer from {{RemoteInputChannel}}
for reading messages from producer.
> The related works are:
> * Add the backlog of producer in {{BufferResponse}} message structure
> * {{PartitionRequestClientHandler}} requests buffer from {{RemoteInputChannel}} directly
> * {{PartitionRequestClientHandler}} updates backlog for {{RemoteInputChannel}}, and it
may trigger requests of floating buffers from {{BufferPool}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message