hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3990) NN's health report has severe performance problems
Date Tue, 16 Oct 2012 14:43:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477053#comment-13477053

Daryn Sharp commented on HDFS-3990:

bq. Maintaining both an ipAddr/hostName plus nodeAddr with the same information, which can
become inconsistent is error prone. For example what do you do when the ipAddr and the nodeAddr

They should never disagree because the nodeAddr is based on the ipAddr, and when the nodeAddr
is changed, so is the ipAddr.

bq. The ipAddr field for a DataNode ID should never change because it (and the xferPort) are
the unique key for a DataNode.

They will change when a pre-existing node, say one with the same storage id, is updated with
the new info.

bq.  We also now have to worry about the state where we're both resolved and unresolved.

We need to worry about that case just like the code did before.  Let's say the exclude list
has hostnames.  A node registration occurs but there's a dns hiccup so all we have is its
ip.  Your proposed patch may let the node in whereas the existing code (and my patch) will
block the node.

bq.  What do you think of the attached patch? It sets the DatanodeID hostname field at registration
time (like the IP addr) ...

The patch appears to change the way the include and exclude work by trusting who the datanode
claims to be.  What if a datanode "lies" about who it is?  Or if a dns hiccup occurs when
the datanode is going to register?  It sends its name as an ip, but the exclude list only
has hosts.  There are a number of scenarios where a datanode could bypass the include/exclude
list, which is why we should never trust the client.

bq. ... using the same lookup we do today and replaces the two problematic lookups with uses
of this field.

Unless I've overlooked something, there's only one lookup that occurs?

I'll post a minor rev for consideration that should further ensure the fields never go out
of sync.
> NN's health report has severe performance problems
> --------------------------------------------------
>                 Key: HDFS-3990
>                 URL: https://issues.apache.org/jira/browse/HDFS-3990
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>         Attachments: HDFS-3990.patch, HDFS-3990.patch, hdfs-3990.txt
> The dfshealth page will place a read lock on the namespace while it does a dns lookup
for every DN.  On a multi-thousand node cluster, this often results in 10s+ load time for
the health page.  10 concurrent requests were found to cause 7m+ load times during which time
write operations blocked.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message