From hdfs-issues-return-12811-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 10 19:30:07 2010 Return-Path: Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: (qmail 26552 invoked from network); 10 Nov 2010 19:30:07 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 10 Nov 2010 19:30:07 -0000 Received: (qmail 36052 invoked by uid 500); 10 Nov 2010 19:30:38 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 35951 invoked by uid 500); 10 Nov 2010 19:30:38 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 35943 invoked by uid 99); 10 Nov 2010 19:30:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Nov 2010 19:30:38 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Nov 2010 19:30:38 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id oAAJUH98001002 for ; Wed, 10 Nov 2010 19:30:17 GMT Message-ID: <19914130.11761289417417610.JavaMail.jira@thor> Date: Wed, 10 Nov 2010 14:30:17 -0500 (EST) From: "Dmytro Molkov (JIRA)" To: hdfs-issues@hadoop.apache.org Subject: [jira] Updated: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1071: -------------------------------- Attachment: HDFS-1071.7.patch Got rid of imageIndex, import of URI in the test case. Converted MiniDFSCluster initialization to use builder Synchronizing on sds when processing IOErrors Should be good now, Jakob can you take another look at this one, thanks. > savenamespace should write the fsimage to all configured fs.name.dir in parallel > -------------------------------------------------------------------------------- > > Key: HDFS-1071 > URL: https://issues.apache.org/jira/browse/HDFS-1071 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node > Reporter: dhruba borthakur > Assignee: Dmytro Molkov > Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.4.patch, HDFS-1071.5.patch, HDFS-1071.6.patch, HDFS-1071.7.patch, HDFS-1071.patch > > > If you have a large number of files in HDFS, the fsimage file is very big. When the namenode restarts, it writes a copy of the fsimage to all directories configured in fs.name.dir. This takes a long time, especially if there are many directories in fs.name.dir. Make the NN write the fsimage to all these directories in parallel. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.