hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Casey Brotherton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
Date Wed, 23 Sep 2015 13:11:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904479#comment-14904479
] 

Casey Brotherton commented on HDFS-9100:
----------------------------------------

Created a multi-node cluster with multi-homed networks.

Created one of the nodes with only one ip address, on the subnet
that was not the primary network of the other hosts in the cluster.

Verified that without appropriate configuration, the lone node 
could not send data to datanodes.

Confirmed that changing dfs.client.use.datanode.hostname allowed the 
lone node to put and read from the datanodes.

Placed data within the HDFS cluster in an unbalanced way.  ( dfs.replication=1 on one datanode,

with minimal data spread across two datanodes with dfs.replication=2)

Running the balancer from the lone node causes these errors:
15/09/23 06:07:04 WARN balancer.Dispatcher: Failed to move blk_1073742600_1776 with size=4128368
from 10.17.74.156:50010:DISK to 10.17.74.158:50010:DISK through 10.17.74.156:50010: Network
is unreachable

After the changes, the balancer worked.  The logging still showed the IP address.
15/09/23 06:05:38 INFO balancer.Dispatcher: Successfully moved blk_1073742139_1315 with size=4128368
from 10.17.74.156:50010:DISK to 10.17.74.158:50010:DISK through 10.17.74.156:50010

This did not test a large environment, where blocks were moved by proxy, although I don't
believe that would cause any problems.


> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---------------------------------------------------------------
>
>                 Key: HDFS-9100
>                 URL: https://issues.apache.org/jira/browse/HDFS-9100
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover, HDFS
>            Reporter: Yongjun Zhang
>            Assignee: Casey Brotherton
>         Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>    private void dispatch() {
>       LOG.info("Start moving " + this);
>       Socket sock = new Socket();
>       DataOutputStream out = null;
>       DataInputStream in = null;
>       try {
>         sock.connect(
>             NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
>             HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of dfs.client.use.datanode.hostname
setting, this would possibly fail balancer run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message