hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3988) The elephant should remember names, not numbers.
Date Thu, 21 Aug 2008 15:16:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624392#action_12624392

Steve Loughran commented on HADOOP-3988:

1. The JVM does lots and lots of DNS caching of its own. To get it to cache positive and negative
DNS entries for less time than forever, you've got to start the processes with a DNS TTL property,
By default, Java5 caches forever: http://java.sun.com/j2se/1.5.0/docs/api/java/net/InetAddress.html
Java6 is a bit smarter, and only caches forever when a security manager is installed

You'd need to decide what is a good DNS cache TTL for the datanodes and push it out to the
scripts. I'm not 100% sure you can set this property inside the JVM and have it taken up;
the command line is the conventional way to do it.

2. The address of the namenode is currently set up in DataNode.startDataNode(). There are
a few classes that assume that DataNode.getNameNodeAddr() is never null; they'd need to change
their assumptions.

3. DNS is slow and is one of the main delays in test runs right now.

What may work is leaving the current init code as it is, but whenever a connection to the
namenode fails, the datanode should re-read the namenode address from the configuration and
redo the nslookup; the scripts would need patching to set a low TTL on the live systems. Rereading
the address from the configuration would be useful if the configuration was coming from something
like an LDAP server; you could change the hostname there and have it picked up without DNS
caching interfering. 

> The elephant should remember names, not numbers.
> ------------------------------------------------
>                 Key: HADOOP-3988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3988
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.2
>            Reporter: Allen Wittenauer
> The name node and the data node should not cache the resolution of host names, as doing
so prevents the use of DNS CNAMEs for any sort of fail over capability.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message