Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 70740 invoked from network); 14 Mar 2007 19:56:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 14 Mar 2007 19:56:40 -0000 Received: (qmail 43716 invoked by uid 500); 14 Mar 2007 19:56:42 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 43675 invoked by uid 500); 14 Mar 2007 19:56:41 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 43458 invoked by uid 99); 14 Mar 2007 19:56:40 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Mar 2007 12:56:40 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Mar 2007 12:56:30 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 285E6714084 for ; Wed, 14 Mar 2007 12:56:10 -0700 (PDT) Message-ID: <8319861.1173902170163.JavaMail.jira@brutus> Date: Wed, 14 Mar 2007 12:56:10 -0700 (PDT) From: "dhruba borthakur (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Updated: (HADOOP-1117) DFS Scalability: When the namenode is restarted it consumes 80% CPU In-Reply-To: <2190936.1173826449231.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HADOOP-1117: ------------------------------------- Attachment: CpuPendingTransfer3.patch Removed some logging messages from previous patch. > DFS Scalability: When the namenode is restarted it consumes 80% CPU > ------------------------------------------------------------------- > > Key: HADOOP-1117 > URL: https://issues.apache.org/jira/browse/HADOOP-1117 > Project: Hadoop > Issue Type: Bug > Components: dfs > Affects Versions: 0.12.0 > Reporter: dhruba borthakur > Assigned To: dhruba borthakur > Priority: Blocker > Fix For: 0.12.1 > > Attachments: CpuPendingTransfer3.patch > > > When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.