hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3941) Extend FileSystem API to return file-checksums/file-digests
Date Tue, 26 Aug 2008 10:13:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625689#action_12625689

Doug Cutting commented on HADOOP-3941:

> Below is a summary of the default getFileChecksum() implementation options [ ... ]

The default should minimize code duplication, if possible.  An abstract method should only
be used for mandatory methods.  Since this is an optional method, a default implementation
should be provided.

The choice of an exception or null depends on the expected use.  An exception should be thrown
for unusual situations that are best handled non-locally, somewhere above the call.  The absence
of a checksum should probably be handled at the site of the call, so returning null seems
a better choice than an exception here.  Another option might be to return a trivial checksum,
e.g., the file's length.

Perhaps we should include a use of this new feature in the patch, to better guide its implementation.
 Should we extend distcp to use this?  Or do you have another canonical application in mind?
 If we add features without applications of them, we risk a design that does not meet any

> Extend FileSystem API to return file-checksums/file-digests
> -----------------------------------------------------------
>                 Key: HADOOP-3941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3941
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: 3941_20080818.patch, 3941_20080819.patch, 3941_20080819b.patch,
> Suppose we have two files in two locations (may be two clusters) and these two files
have the same size.  How could we tell whether the content of them are the same?
> Currently, the only way is to read both files and compare the content of them.  This
is a very expensive operation if the files are huge.
> So, we would like to extend the FileSystem API to support returning file-checksums/file-digests.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message