hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4075) Reduce recommissioning overhead
Date Wed, 24 Oct 2012 13:50:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483229#comment-13483229

Kihwal Lee commented on HDFS-4075:

bq. Can you please also explain the change in DatanodeManager.java in this patch? node.isAlive
will be updated only when the node heartbeats in. So when will blockManager.processOverReplicatedBlocksOnReCommission(node);
be called?

When a node registers and sends in its first full block report, over-replicated blocks will
be taken care of. So it is wasteful to go over the blocks owned by dead nodes on recommission.

I will revisit this issue after HDFS-3937 and HDFS-4080.
> Reduce recommissioning overhead
> -------------------------------
>                 Key: HDFS-4075
>                 URL: https://issues.apache.org/jira/browse/HDFS-4075
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.23.4, 2.0.2-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-4075.patch
> When datanodes are recommissioned, {BlockManager#processOverReplicatedBlocksOnReCommission()}
is called for each rejoined node and excess blocks are added to the invalidate list. The problem
is this is done while the namesystem write lock is held.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message