hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode is restarted it consumes 80% CPU
Date Sat, 14 May 2011 23:48:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033619#comment-13033619
] 

Eli Collins commented on HADOOP-1117:
-------------------------------------

Doesn't this break the rack policy? We should only remove a block from neededReplications
when it has sufficient replicas both in terms of replica count and rack count.

{noformat}
+        if (numCurrentReplica >= fileReplication) {
+          neededReplications.remove(block);
+        } else 
{noformat}

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into
neededReplication. When the namenode exists, safemode it sees starts processing neededReplication.
It picks up a block from neededReplication, sees that it has already has the required number
of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication
permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds.
This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message