hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5467) Create an offline fsimage image viewer
Date Fri, 27 Mar 2009 01:32:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689778#action_12689778

Konstantin Shvachko commented on HADOOP-5467:

This is a very useful tool.
My main concern here is that this tool introduces an alternative fsimage loader (reader),
which may easily become out of sync with the loader we have in FSImage class.
Ideally we should have the same source code reading the fsimage file and then using different
visitors to process deserialized data. I think we can achieve that goal by implementing a
{{LoadFSImageVisitor}}, which will call FSNamesystem methods to add inodes to the directory
tree and so on, making it a replacement to {{FSImage.loadFSImage()}}.
The {{LoadFSImageVisitor}} can be passed to {{FSImageProcessor}} same as other visitors Jakob
We can do it in a separate Jira, but it should be done before the next release so that we
had uniform deserialization in the release.
This approach will probably also require to move {{FSImageProcessor}} code inside {{server.namenode}}
package. The {{OfflineImageViewer}} itself should remain in tools.

Other comments:
# offlineimageviewer should be a part of hdfs shell command group rather than hadoop.
# I would shorten it to just imageviewer.
# When I call {{hadoop offlineimageviewer}} it first prints an error: "Error parsing options:
# option "-o" does not work together with "-p XML". Please check other combination too.
# OfflineImageViewer.java warnings in line 135 about accessing static methods in non static
# {{FSImageProcessorV16to19}} should be renamed to something more version independent.
If you go to v -20 you will probably modify this class rather than implement a new one.
I'd probably rename the interface {{FSImageProcessor}} into {{FSImageProcessorInterface}}
and then
{{FSImageProcessorV16to19}} can be renamed to {{FSImageProcessor}}.
# {{if ( p.canProcessVersion(version) )}} should not have spaces after "(" and before ")".
# {{TextWriterFSImageProcessor}} should probably be {{TextWriterFSImageVisitor}}.
# {{FSImageElement}} should be declared in {{FSImageVisitor}}.
# We do not want to use deprecated UTF8 class more than it is used already, so it is better
use {{FSImage.readBytes()}}, etc. instead of reimplementing them in {{FSImageProcessor}}.

> Create an offline fsimage image viewer
> --------------------------------------
>                 Key: HADOOP-5467
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5467
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: fsimage.xml, HADOOP-5467.patch, HADOOP-5467.patch, HADOOP-5467.patch
> It would be useful to have a tool to examine/dump the contents of the fsimage file to
human-readable form.  This would allow analysis of the namespace (file usage, block sizes,
etc) without impacting the operation of the namenode.  XML would be reasonable output format,
as it can be easily viewed, compressed and manipulated via either XSLT or XQuery.  
> I've started work on this and will have an initial version soon.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message