hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lin Yiqun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9375) Set balancer bandwidth for specific node
Date Tue, 10 Nov 2015 08:10:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998226#comment-14998226
] 

Lin Yiqun commented on HDFS-9375:
---------------------------------

Thanks [~brahmareddy] review my patch.I address most of your comment.But there is a problem
that even though {{ClientDatanodeProtocol}} has no method {{setBalancerBandwidth}},but {{ClientNamenodeProtocol}}
has, and it will lead to a protoc compiler error.The reason is below
{code}
[DEBUG] ClientNamenodeProtocol.proto:637:9: "hadoop.hdfs.SetBalancerBandwidthRequestProto"
is already defined in file "ClientDatanodeProtocol.proto".
[DEBUG] ClientNamenodeProtocol.proto:641:9: "hadoop.hdfs.SetBalancerBandwidthResponseProto"
is already defined in file "ClientDatanodeProtocol.proto".
[DEBUG] ClientNamenodeProtocol.proto:808:28: "hadoop.hdfs.SetBalancerBandwidthRequestProto"
seems to be defined in "ClientDatanodeProtocol.proto", which is not imported by "ClientNamenodeProtocol.proto".
 To use it here, please add the necessary import.
[DEBUG] ClientNamenodeProtocol.proto:809:15: "hadoop.hdfs.SetBalancerBandwidthResponseProto"
seems to be defined in "ClientDatanodeProtocol.proto", which is not imported by "ClientNamenodeProtocol.proto".
 To use it here, please add the necessary import.
[ERROR] protoc compiler error
{code}
So I define a new method name {{ClientDatanodeProtocol#setBalanceThrottlerBandwidth}},and
because this is set for datanode's Throttler bandwidth.And I attach the updated patch.

> Set balancer bandwidth for specific node
> ----------------------------------------
>
>                 Key: HDFS-9375
>                 URL: https://issues.apache.org/jira/browse/HDFS-9375
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>    Affects Versions: 2.7.1
>            Reporter: Lin Yiqun
>            Assignee: Lin Yiqun
>         Attachments: HDFS-9375.001.patch, HDFS-9375.002.patch, HDFS-9375.003.patch
>
>
> Now even though the balancer is impove a lot, but In some cases that is still slow. For
example when the cluster is extended,the new nodes all need balancer datas from existed nodes.In
order to improve the balancer velocity,generally,we will use {{setBalancerBandwidth}} of {{dfsadmin}}
command.But this is set for every node, obviously,we can increase more bandwidth for new nodes
because these nodes lacking of data.When the new nodes balancer data enough,we can let new
nodes to work.So we can define a new clientDatanode interface to set specific node's bandwidth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message