Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1837627121.1240880250854.JavaMail.jira@brutus>
Date: Mon, 27 Apr 2009 17:57:30 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5752) Provide examples of using offline
 image viewer (oiv) to analyze hadoop file systems
In-Reply-To: <458471969.1240879770453.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-5752:
--------------------------------

    Attachment: HADOOP-5752.patch

The OIV's output data are ripe for analysis.  The attached patch:
* Creates a new image processor, Delimited, that creates a (by default) tab-delimited file of the namespace that is suitable for analysis by other tools. 
* Updates the the oiv documentation to provide examples of how to analyze these files using Pig to find probable duplicate files, files that have never been accessed and the total number of files of each user in the namespace. These are meant as examples to help ops and such build other useful scripts.
* Provides unit test for new DelimitedImageVisitor

Right now the script files themselves are not included in the patch because I couldn't figure out a good place to stash them in the file structure.  Konstantin suggested adding them to the wiki, which would be nice as other users could add other scripts as they are created, but I don't see where the wiki hosts files like these.  If it can, can someone please point me to them?

Santhosh from the Pig team kindly reviewed and blessed the pig scripts.

> Provide examples of using offline image viewer (oiv) to analyze hadoop file systems
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-5752
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5752
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: HADOOP-5752.patch
>
>
> The offline image viewer provides the ability to generate large amounts of data about an hdfs namespace. It would be good to provide tools, examples, etc. on how to analyze this data to find useful information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.