From hdfs-dev-return-1589-apmail-hadoop-hdfs-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 28 21:44:57 2010 Return-Path: Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: (qmail 64762 invoked from network); 28 Sep 2010 21:44:57 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Sep 2010 21:44:57 -0000 Received: (qmail 73976 invoked by uid 500); 28 Sep 2010 21:44:56 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 73777 invoked by uid 500); 28 Sep 2010 21:44:56 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 73624 invoked by uid 99); 28 Sep 2010 21:44:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 21:44:56 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 21:44:55 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o8SLiZJ8004960 for ; Tue, 28 Sep 2010 21:44:35 GMT Message-ID: <24328773.451671285710275055.JavaMail.jira@thor> Date: Tue, 28 Sep 2010 17:44:35 -0400 (EDT) From: "Hairong Kuang (JIRA)" To: hdfs-dev@hadoop.apache.org Subject: [jira] Created: (HDFS-1425) Improve NameNode responsiveness by speeding up serving refreshNodes RPC MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Improve NameNode responsiveness by speeding up serving refreshNodes RPC ----------------------------------------------------------------------- Key: HDFS-1425 URL: https://issues.apache.org/jira/browse/HDFS-1425 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: 0.22.0 Currently when NN serves the refreshNodes request, it examines every block in the nodes to be decommissioned and put it in the neededReplication queue, during which a write lock is held. In our production cluster when decommissioning 100 nodes, we observed that refreshingNodes took minutes, during the period NameNode became non responsive. The proposal is that freshNodes only adds the nodes that need to be decommissioned in a queue in DecommissionManager and then return. DecommissionManager then takes care of decommissioning the nodes. This would allow refreshNodes to be returned very quickly. This also allows us to optimize DecommisionManager. For example, it could sleep when there is no datanode to be decommissioned while the current code wakes the thread periodically even when no decommission in progress nodes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.