hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4269) DatanodeManager#registerDatanode rejects all datanode registrations from localhost in single-node developer setup
Date Tue, 23 Apr 2013 15:13:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639143#comment-13639143

Daryn Sharp commented on HDFS-4269:

[~jagane] I understand your frustration, but regarding the presumed "refuctoring": as far
as I can tell, it was a refactoring.  I reviewed my change in HDFS-3990.  I didn't alter the
handling of the allow/deny lists.

The original implementation always did a reverse lookup in {{DatanodeManager.checkInList}}
– this caused massive dns lookups for the web pages and new node registrations.  {{checkInList}}
when called via {{inHostsList}} returned false if the host was unresolvable.  Within {{DatanodeManager#registerDatanode}},
the check {{!inHostsList(nodeReg)}} would throw a {{DisallowedDatanodeException}}.

All I did was hoist the reverse lookup from {{checkInList}} up into {{registerDatanode}},
and store the resolved hostname for later use by {{checkInList}} to prevent unnecessary reverse
lookups.  The localhost bug was introduced because I accommodated tests faking the hostname.
 I compared the ip and hostname instead of checking {{InetSocketAddress#isUnresolved}}.  On
non-windows systems, resolves to localhost; windows wrongly claims resolves
to, hence it appeared unresolved.

Regarding your points:
# Not requiring hosts to resolve if there are only IPs in _both_ lists may be feasible.
# If you have multiple A records for your datanodes that confuse your browser, I think the
{{DFSClient}} will be equally confused.
# Long ago, the NN operated only on IPs.  It was changed to support grids using public IPs,
and a private intra-grid network with different IPs, where the public and private networks
resolved a hostname to the correct IP.
# The 'does reverse lookup succeed' cannot be only for exclusion.  Decommissioning is signified
by the host being in both lists.
> DatanodeManager#registerDatanode rejects all datanode registrations from localhost in
single-node developer setup
> -----------------------------------------------------------------------------------------------------------------
>                 Key: HDFS-4269
>                 URL: https://issues.apache.org/jira/browse/HDFS-4269
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 3.0.0, trunk-win, 2.0.5-beta
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 3.0.0
>         Attachments: HDFS-4269.1.patch, HDFS-4269.2.patch, HDFS-4269.3.patch
> HDFS-3990 is a change that optimized some redundant DNS lookups.  As part of that change,
{{DatanodeManager#registerDatanode}} now rejects attempts to register a datanode for which
the name has not been resolved.  Unfortunately, this broke single-node developer setups on
Windows, because Windows does not resolve to "localhost".

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message