hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Molkov (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-599) Improve Namenode robustness by prioritizing datanode heartbeats over client requests
Date Fri, 14 May 2010 17:56:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867600#action_12867600
] 

Dmytro Molkov commented on HDFS-599:
------------------------------------

Hairong, thanks for your comments.

Not starting RPC client server until we are out of safemode is the second patch that we have
been running internally for a while now and I will port it to trunk as soon as this jira makes
it in. I felt like adding both parts in one jira will be too huge.

DFSAdmin in that case does have to run on the service port.

The clean separation makes sense, but I do not think we can fully make that separation available
in the case of this JIRA.
The way to effectively administratively solve the problem of two ports is to firewall service
ports from external clients + do not include information about this port in the mapreduce
configuration. This way only HDFS cluster will have the information in the configuration and
only the datanodes will be accessing it. This is the way we were operating internally at FB.

This of course doesn't help solve the problem of malicious clients still accessing the service
port by hacking the values in the code (since it should not be available in the configuration).

However removing the ClientProtocol from the service port will effectively make it impossible
for administrator to perform any client operations like LS, or even getting out of safemode
(which is still in ClientProtocol) if we postpone the start of the client port until we are
out of safemode.

So essentially I feel like this problem can partly be solved by administrative measures and
the value that we get from keeping the Client protocol and others available on the service
port still outweigh the problem of malicious clients that might get in on that port.

> Improve Namenode robustness by prioritizing datanode heartbeats over client requests
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-599
>                 URL: https://issues.apache.org/jira/browse/HDFS-599
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HDFS-599.patch
>
>
> The namenode processes RPC requests from clients that are reading/writing to files as
well as heartbeats/block reports from datanodes.
> Sometime, because of various reasons (Java GC runs, inconsistent performance of NFS filer
that stores HDFS transacttion logs, etc), the namenode encounters transient slowness. For
example, if the device that stores the HDFS transaction logs becomes sluggish, the Namenode's
ability to process RPCs slows down to a certain extent. During this time, the RPCs from clients
as well as the RPCs from datanodes suffer in similar fashion. If the underlying problem becomes
worse, the NN's ability to process a heartbeat from a DN is severly impacted, thus causing
the NN to declare that the DN is dead. Then the NN starts replicating blocks that used to
reside on the now-declared-dead datanode. This adds extra load to the NN. Then the now-declared-datanode
finally re-establishes contact with the NN, and sends a block report. The block report processing
on the NN is another heavyweight activity, thus casing more load to the already overloaded
namenode. 
> My proposal is tha the NN should try its best to continue processing RPCs from datanodes
and give lesser priority to serving client requests. The Datanode RPCs are integral to the
consistency and performance of the Hadoop file system, and it is better to protect it at all
costs. This will ensure that NN  recovers from the hiccup much faster than what it does now.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message