hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9209) Add shell command to dump file checksums
Date Tue, 15 Jan 2013 02:14:12 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HADOOP-9209:

    Attachment: hadoop-9209.txt

Attached patch implements the new shell command.

In addition to the unit test, I tested manually:

$ ./bin/hadoop fs -checksum '/*'
/file1  MD5-of-0MD5-of-512CRC32C        000002000000000000000000b234aa05a75fed38536bda657b20bfcf
/file1-crc32        MD5-of-0MD5-of-512CRC32 000002000000000000000000593b23e67a7477aab90e42e41478b321
/file1-crc32-copy   MD5-of-0MD5-of-512CRC32 000002000000000000000000593b23e67a7477aab90e42e41478b321

$ ./bin/hadoop fs -help checksum
-checksum <src> ...:    Dump checksum information for files that match the file
                pattern <src> to stdout. Note that this requires a round-trip
                to the datanode storing each block of the file, and thus is not
                efficient to run on a large number of files.
> Add shell command to dump file checksums
> ----------------------------------------
>                 Key: HADOOP-9209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9209
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, tools
>    Affects Versions: 3.0.0, 2.0.3-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-9209.txt
> Occasionally while working with tools like distcp, or debugging certain issues, it's
useful to be able to quickly see the checksum of a file. We currently have the APIs to efficiently
calculate a checksum, but we don't expose it to users. This JIRA is to add a "fs -checksum"
command which dumps the checksum information for the specified file(s).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message