hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9209) Add shell command to dump file checksums
Date Tue, 15 Jan 2013 02:14:12 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HADOOP-9209:
--------------------------------

    Attachment: hadoop-9209.txt

Attached patch implements the new shell command.

In addition to the unit test, I tested manually:

{code}
$ ./bin/hadoop fs -checksum '/*'
/file1  MD5-of-0MD5-of-512CRC32C        000002000000000000000000b234aa05a75fed38536bda657b20bfcf
/file1-crc32        MD5-of-0MD5-of-512CRC32 000002000000000000000000593b23e67a7477aab90e42e41478b321
/file1-crc32-copy   MD5-of-0MD5-of-512CRC32 000002000000000000000000593b23e67a7477aab90e42e41478b321

$ ./bin/hadoop fs -help checksum
-checksum <src> ...:    Dump checksum information for files that match the file
                pattern <src> to stdout. Note that this requires a round-trip
                to the datanode storing each block of the file, and thus is not
                efficient to run on a large number of files.
{code}
                
> Add shell command to dump file checksums
> ----------------------------------------
>
>                 Key: HADOOP-9209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9209
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, tools
>    Affects Versions: 3.0.0, 2.0.3-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-9209.txt
>
>
> Occasionally while working with tools like distcp, or debugging certain issues, it's
useful to be able to quickly see the checksum of a file. We currently have the APIs to efficiently
calculate a checksum, but we don't expose it to users. This JIRA is to add a "fs -checksum"
command which dumps the checksum information for the specified file(s).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message