hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1875) MiniDFSCluster hard-codes dfs.datanode.address to localhost
Date Thu, 26 May 2011 00:41:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039455#comment-13039455

Matt Foley commented on HDFS-1875:

Hi Eric, looks generally good.  A few comments:

# MiniDFSCluster
** startDataNodes() - the "@param checkDataNodeAddrConfig" comment has been added to the javadocs
for the stubbed-out original method signature.  It needs to be associated with the new longer-args-list
method signature.

# TestDFSAddressConfig
** Configuration.unset() method is only available in trunk, not yahoo-merge, at this time.
 This was added to trunk in HADOOP-7001 on Nov 24, 2010, but seems to not be in yahoo-merge.
 Have you asked Suresh to merge HADOOP-7001 to yahoo-merge?

# DataNodeCluster
** In the class javadocs and/or the USAGE string, should document the change in semantics
- that the datanode addresses specified in config will be used unless not set.

Also, with this setup, if you then run the Client on the same server as the DataNodeCluster,
can the Client communicate successfully?  Or does it try to use and fail to connect?
 Have you searched for other unit test classes that use a Client and a DataNodeCluster?  Do
they all still work?


> MiniDFSCluster hard-codes dfs.datanode.address to localhost
> -----------------------------------------------------------
>                 Key: HDFS-1875
>                 URL: https://issues.apache.org/jira/browse/HDFS-1875
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>             Fix For: 0.23.0
>         Attachments: HDFS-1875.patch
> When creating RPC addresses that represent the communication sockets for each simulated
DataNode, the MiniDFSCluster class hard-codes the address of the dfs.datanode.address port
to be ""
> The DataNodeCluster test tool uses the MiniDFSCluster class to create a selected number
of simulated datanodes on a single host. In the DataNodeCluster setup, the NameNode is not
simulated but is started as a separate daemon.
> The problem is that if the write requrests into the simulated datanodes are originated
on a host that is not the same host running the simulated datanodes, the connections are refused.
This is because the RPC sockets that are started by MiniDFSCluster are for "localhost" (
and are not accessible from outside that same machine.
> It is proposed that the MiniDFSCluster.setupDatanodeAddress() method be overloaded in
order to accommodate an environment where the NameNode is on one host, the client is on another
host, and the simulated DataNodes are on yet another host (or even multiple hosts simulating
multiple DataNodes each).
> The overloaded API would add a parameter that would be used as the basis for creating
the RPS sockets. By default, it would remain

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message