hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-928) make checksums optional per FileSystem
Date Tue, 27 Feb 2007 19:13:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476311
] 

Doug Cutting commented on HADOOP-928:
-------------------------------------

The sum buffer should probably be no smaller than the default and also should be flushed no
more frequently than the main buffer.  So that would indicate that it be sized to the larger
of io.file.buffer.size and bufferSize/bytesPerSum.  Does that sound right to you?

> make checksums optional per FileSystem
> --------------------------------------
>
>                 Key: HADOOP-928
>                 URL: https://issues.apache.org/jira/browse/HADOOP-928
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Doug Cutting
>         Assigned To: Hairong Kuang
>         Attachments: checksum.patch, checksum1.patch, checksum2.patch
>
>
> Checksumming is currently built into the base FileSystem class.  It should instead be
optional, with each FileSystem implementation electing whether to use the Hadoop-provided
checksum system, or to disable it, or to implement its own custom checksum system.
> To implement this, a ChecksumFileSystem implementation can be provided that wraps another
FileSystem implementation, implementing checksums as in Hadoop's current mandatory implementation
(i.e., as a separate crc file per file that's elided from directory listings).  The 'raw'
FileSystem methods would be removed.  FSDataInputStream and FSDataOutputStream would be made
interfaces.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message