Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5DB8A10873 for ; Wed, 5 Jun 2013 23:46:24 +0000 (UTC) Received: (qmail 80940 invoked by uid 500); 5 Jun 2013 23:46:23 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 80897 invoked by uid 500); 5 Jun 2013 23:46:23 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 80810 invoked by uid 99); 5 Jun 2013 23:46:23 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jun 2013 23:46:23 +0000 Date: Wed, 5 Jun 2013 23:46:22 +0000 (UTC) From: "Hudson (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-4850) fix OfflineImageViewer to work on fsimages with empty files or snapshots MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676503#comment-13676503 ] Hudson commented on HDFS-4850: ------------------------------ Integrated in Hadoop-trunk-Commit #3866 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3866/]) HDFS-4850. Fix OfflineImageViewer to work on fsimages with empty files or snapshots. Contributed by Jing Zhao. (Revision 1490080) Result = SUCCESS jing9 : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1490080 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TextWriterImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/XmlImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java > fix OfflineImageViewer to work on fsimages with empty files or snapshots > ------------------------------------------------------------------------ > > Key: HDFS-4850 > URL: https://issues.apache.org/jira/browse/HDFS-4850 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools > Reporter: Stephen Chu > Assignee: Jing Zhao > Labels: snapshot > Attachments: datadirs.tar.gz, fsimage_0000000000000000004, fsimage_0000000000000000008, HDFS-4850.000.patch, HDFS-4850.001.patch, oiv_out_1, oiv_out_2 > > > I deployed hadoop-trunk HDFS and created _/user/schu/_. I then forced a checkpoint, fetched the fsimage, and ran the default OfflineImageViewer successfully on the fsimage. > {code} > schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000004 -o oiv_out_1 > schu-mbp:~ schu$ cat oiv_out_1 > drwxr-xr-x - schu supergroup 0 2013-05-24 16:59 / > drwxr-xr-x - schu supergroup 0 2013-05-24 16:59 /user > drwxr-xr-x - schu supergroup 0 2013-05-24 16:59 /user/schu > schu-mbp:~ schu$ > {code} > I then touched an empty file _/user/schu/testFile1_ > {code} > schu-mbp:~ schu$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - schu supergroup 0 2013-05-24 16:59 /user > drwxr-xr-x - schu supergroup 0 2013-05-24 17:00 /user/schu > -rw-r--r-- 1 schu supergroup 0 2013-05-24 17:00 /user/schu/testFile1 > {code} > and forced another checkpoint, fetched the fsimage, and reran the OfflineImageViewer. I encountered a NegativeArraySizeException: > {code} > schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000008 -o oiv_out_2 > Input ended unexpectedly. > 2013-05-24 17:01:13,622 ERROR [main] offlineImageViewer.OfflineImageViewer (OfflineImageViewer.java:go(140)) - image loading failed at offset 402 > Exception in thread "main" java.lang.NegativeArraySizeException > at org.apache.hadoop.io.Text.readString(Text.java:458) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processPermission(ImageLoaderCurrent.java:370) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINode(ImageLoaderCurrent.java:671) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processChildren(ImageLoaderCurrent.java:557) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:464) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processLocalNameINodesWithSnapshot(ImageLoaderCurrent.java:444) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINodes(ImageLoaderCurrent.java:398) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:199) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:136) > at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.main(OfflineImageViewer.java:260) > {code} > This is reproducible. I've reproduced this scenario after formatting HDFS and restarting and touching an empty file _/testFile1_. > Attached are the data dirs, the fsimage before creating the empty file (fsimage_0000000000000000004) and the fsimage afterwards (fsimage_0000000000000000004) and their outputs, oiv_out_1 and oiv_out_2 respectively. > The oiv_out_2 does not include the empty _/user/schu/testFile1_. > I don't run into this problem using hadoop-2.0.4-alpha. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira