Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0A7B1753B for ; Mon, 29 Sep 2014 15:51:50 +0000 (UTC) Received: (qmail 10353 invoked by uid 500); 29 Sep 2014 15:51:50 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 10283 invoked by uid 500); 29 Sep 2014 15:51:50 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 10274 invoked by uid 99); 29 Sep 2014 15:51:50 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Sep 2014 15:51:50 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 2BB459C23F5; Mon, 29 Sep 2014 15:51:50 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: aw@apache.org To: common-commits@hadoop.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: git commit: HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw) Date: Mon, 29 Sep 2014 15:51:50 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/trunk 4666440b1 -> 9829f9fa5 HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9829f9fa Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9829f9fa Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9829f9fa Branch: refs/heads/trunk Commit: 9829f9fa58bbc8b4237d3182ee7f4eff5487aa52 Parents: 4666440 Author: Allen Wittenauer Authored: Mon Sep 29 08:51:41 2014 -0700 Committer: Allen Wittenauer Committed: Mon Sep 29 08:51:41 2014 -0700 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 2 + .../src/main/resources/hdfs-default.xml | 11 ++++ .../src/site/apt/HdfsImageViewer.apt.vm | 59 +++++++++++++++++++- 3 files changed, 71 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/9829f9fa/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 5e1d717..767e11f0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -516,6 +516,8 @@ Release 2.6.0 - UNRELEASED HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal) + HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw) + OPTIMIZATIONS HDFS-6690. Deduplicate xattr names in memory. (wang) http://git-wip-us.apache.org/repos/asf/hadoop/blob/9829f9fa/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml index 23b25fa..1cc68f4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml @@ -2140,4 +2140,15 @@ + + dfs.namenode.legacy-oiv-image.dir + + Determines where to save the namespace in the old fsimage format + during checkpointing by standby NameNode or SecondaryNameNode. Users can + dump the contents of the old format fsimage by oiv_legacy command. If + the value is not specified, old format fsimage will not be saved in + checkpoint. + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/9829f9fa/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm index 9a9946e..3b84226 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm @@ -28,7 +28,7 @@ Offline Image Viewer Guide namespace. The tool is able to process very large image files relatively quickly. The tool handles the layout formats that were included with Hadoop versions 2.4 and up. If you want to handle older layout formats, you can - use the Offline Image Viewer of Hadoop 2.3. + use the Offline Image Viewer of Hadoop 2.3 or {{oiv_legacy Command}}. If the tool is not able to process an image file, it will exit cleanly. The Offline Image Viewer does not require a Hadoop cluster to be running; it is entirely offline in its operation. @@ -188,3 +188,60 @@ Offline Image Viewer Guide about the hdfs namespace. This information can then be used to explore file system usage patterns or find specific files that match arbitrary criteria, along with other types of namespace analysis. + +* oiv_legacy Command + + Due to the internal layout changes introduced by the ProtocolBuffer-based + fsimage ({{{https://issues.apache.org/jira/browse/HDFS-5698}HDFS-5698}}), + OfflineImageViewer consumes excessive amount of memory and loses some + functions such as Indented and Delimited processor. If you want to process + without large amount of memory or use these processors, you can use + <<>> command (same as <<>> in Hadoop 2.3). + +** Usage + + 1. Set <<>> to an appropriate directory + to make standby NameNode or SecondaryNameNode save its namespace in the + old fsimage format during checkpointing. + + 2. Use <<>> command to the old format fsimage. + +---- + bash$ bin/hdfs oiv_legacy -i fsimage_old -o output +---- + +** Options + +*-----------------------:-----------------------------------+ +| <> | <> | +*-----------------------:-----------------------------------+ +| <<<-i>>>\|<<<--inputFile>>> | Specify the input fsimage file to +| | process. Required. +*-----------------------:-----------------------------------+ +| <<<-o>>>\|<<<--outputFile>>> | Specify the output filename, if +| | the specified output processor generates one. If the +| | specified file already exists, it is silently +| | overwritten. Required. +*-----------------------:-----------------------------------+ +| <<<-p>>>\|<<<--processor>>> | Specify the image processor to +| | apply against the image file. Valid options are +| | Ls (default), XML, Delimited, Indented, and +| | FileDistribution. +*-----------------------:-----------------------------------+ +| <<<-skipBlocks>>> | Do not enumerate individual blocks within files. This +| | may save processing time and outfile file space on +| | namespaces with very large files. The Ls processor +| | reads the blocks to correctly determine file sizes +| | and ignores this option. +*-----------------------:-----------------------------------+ +| <<<-printToScreen>>> | Pipe output of processor to console as well as +| | specified file. On extremely large namespaces, this +| | may increase processing time by an order of +| | magnitude. +*-----------------------:-----------------------------------+ +| <<<-delimiter>>> | When used in conjunction with the Delimited +| | processor, replaces the default tab delimiter with +| | the string specified by . +*-----------------------:-----------------------------------+ +| <<<-h>>>\|<<<--help>>>| Display the tool usage and help information and exit. +*-----------------------:-----------------------------------+