hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
Date Wed, 01 Sep 2010 23:11:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905301#action_12905301
] 

Hairong Kuang commented on HDFS-1348:
-------------------------------------

yes, we should restrict the scanning overhead by putting an up limit on the number of block
fetches. So I would propose to set the number of blocks to be fetched to
  max(2000, total # of blocks/5)
So the one check approximately scans the block list up to 5 times.

As the checkedBlock overhead, I plan to use LightweightGSet introduced by HDFS-1114.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decommission.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But
once for a while, NameNode seems to pause for more than 10 seconds without doing anything,
leaving a blank in its log even though no garbage collection is happening.  All other requests
to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the
whole process of checking if decomissioning DataNodes are finished or not, during which it
checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message