flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhijiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6337) Remove the buffer provider from PartitionRequestServerHandler
Date Tue, 25 Apr 2017 08:49:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982574#comment-15982574

zhijiang commented on FLINK-6337:

Thank you for so quick response! 

Currently the buffer provider only provides the segment size for SpillableSubpartitionView,
and the actual buffers which would be spilled to disk are provided by LocalBufferPool from
ResultPartition. If SpillableSubpartitionView needs another buffer provider  to copy the buffers
from ResultPartition in future plans, I think this buffer provider can be provided from result
partition manager or other components instead of server handler, because the handler can not
get the result partition type. And current mode will waste a lot of available buffers from
global buffer pool. If there are no specific plans and time for this refactoring, remove it
from handler can get benefits and not effect the current behavior.

The motivation is that our users are confused of why the total available buffers are less
than the amount in configuration.

> Remove the buffer provider from PartitionRequestServerHandler
> -------------------------------------------------------------
>                 Key: FLINK-6337
>                 URL: https://issues.apache.org/jira/browse/FLINK-6337
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
> Currently, {{PartitionRequestServerHandler}} will create a {{LocalBufferPool}} when the
channel is registered. The {{LocalBufferPool}} is only used to get segment size for creating
read view in {{SpillableSubpartition}}, and the buffers in the pool will not be used all the
time, so it will waste the buffer resource of global pool.
> We would like to remove the {{LocalBufferPool}} from the {{PartitionRequestServerHandler}},
and the {{LocalBufferPool}} in {{ResultPartition}} can also provide the segment size for creating
sub partition view.

This message was sent by Atlassian JIRA

View raw message