hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7178) copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
Date Sun, 03 Jul 2011 11:56:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059185#comment-13059185

Uma Maheswara Rao G commented on HADOOP-7178:

Hi Todd,

 Thanks for the comments.

I don't agree with the intent of this issue. Adding a new flag to turn on/off the writing
of checksums sounds reasonable – but disabling the writing of checksums based on whether
you asked to verify it on the way out of HDFS seems like an unrelated choice.
 Agree :-). As you said, Using setVerifyChecksum api to control the checksum outs may not
be correct choice.
 To maintain the consistancy, we can provide one API on FileSystem, i.e, fs.ignoreCRC().
 Internally this API should take care of setting the flag setVerifyChecksum to false and also
should select the non CheckSumFileSystem as local file system. I think, this should be fine.

Initially we thought like, why we need to create the checksum out when user don't want to
do any checksum related verification. this may be little bit confusion when we look at the
api setVerifyChecksum. I will update the issue title as well.

Please give your opinion also.

 As i know this particular behavior already available with command line.
 This issue mainly for selecting the local side file system based on the flag. Copying the
checksum outs will be depending on the fileSystem which it selects at that point.

This is handled in command line options by providing the option 'ignoreCrc' this flag also
will set the setVerifyChecksum  to false on FileSystem. 

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to
false at client side.
> -----------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem
for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the
use in creating of these files in local file system.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message