hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-746) CRC computation and reading should move into a nested FileSystem
Date Mon, 04 Dec 2006 08:03:26 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-746?page=comments#action_12455248 ] 
dhruba borthakur commented on HADOOP-746:

Please allow me to see if I can sum up the ideas and try to arrive at a consensus:

1. There is a CRC that is computed and used internally by HDFS. HDFS computes this CRC per
each block of the file. No files with *.crc are created in the namespace. 

2. HDFS exposes an API that allows an application to retrieve the CRC of each block of an
HDFS file.

3. HDFS exposes another API that an application can use to switch off CRC usage and calculation
by HDFS.


> CRC computation and reading should move into a nested FileSystem
> ----------------------------------------------------------------
>                 Key: HADOOP-746
>                 URL: http://issues.apache.org/jira/browse/HADOOP-746
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.8.0
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
> Currently FileSystem provides both an interface and a mechanism for computing and checking
crc files. I propose splitting the crc code into a nestable FileSystem that like the PhasedFileSystem
has a backing FileSystem. Once the Paths are converted to URI, this is fairly natural to express.
To use crc files, your uris will look like:
> crc://hdfs:%2f%2fhost1:8020/ which is a crc FileSystem with an underlying file system
of hdfs://host1:8020
> This will allow users to use crc files where they make sense for their application/cluster
and get rid of the "raw" methods.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message