hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-738) dfs get or copyToLocal should not copy crc file
Date Tue, 21 Nov 2006 23:51:03 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-738?page=comments#action_12451824 ] 
eric baldeschwieler commented on HADOOP-738:

The issues you raise are real when you are moving multi-terabyte search indexes and such.
 Having end-2-end CRC support is crucial and I'm all for keeping it in.  But yes, our average
use is not "sophisticated" in that their average operation to local disk in our environment
is just to pull some experiment results locally.  In general modern machines are good enough
that megabyte to gigabyte sized files can be read and written reliably enough that CRC errors
are not a dominate concern.

Where they are a concern, I suggest we make the CRC files visible.  Rather than using "dot"
files, just append .crc to the file name.  Then at least folks can see them and ask the right
questions, etc.  Things will work better that way.

> dfs get or copyToLocal should not copy crc file
> -----------------------------------------------
>                 Key: HADOOP-738
>                 URL: http://issues.apache.org/jira/browse/HADOOP-738
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.8.0
>         Environment: all
>            Reporter: Milind Bhandarkar
>         Assigned To: Milind Bhandarkar
>             Fix For: 0.9.0
>         Attachments: hadoop-crc.patch
> Currently, when we -get or -copyToLocal a directory from DFS, all the files including
crc files are also copied. When we -put or -copyFromLocal again, since the crc files already
exist on DFS, this put fails. The solution is not to copy checksum files when copying to local.
Patch is forthcoming.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message