hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1392) Improve namenode scalability by prioritizing datanode heartbeats over block reports
Date Sat, 11 Sep 2010 01:25:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908260#action_12908260
] 

dhruba borthakur commented on HDFS-1392:
----------------------------------------

Following the footsteps of HDFS-599, one proposal is to have a yet another separate rpc port
on the namenode that is used by the datanode to send *only* heartbeats. 

> Improve namenode scalability by prioritizing datanode heartbeats over block reports
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-1392
>                 URL: https://issues.apache.org/jira/browse/HDFS-1392
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> When a namenode restarts, it gets heartbeats followed by a block reports from the datanodes.
The block report processing is heavyweight and can take many many seconds, especially when
the number of blocks in each datanode is large. This causes the namenode to not be able to
process heartbeats in time, thus leading to "lost heartbeats from datanode". The namenode
declares the datanode to be dead and the datanode has to re-send its register/block report
all over again. This is a scalability problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message