hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Many Checksum Errors
Date Wed, 16 May 2007 19:07:17 GMT
Raghu Angadi wrote:
> In my implementation of block-level CRCs (does not affect 
> ChecksumFileSystem in HADOOP-928), we don't buffer checksum data at all. 

That sounds like a good approach.  I look forward to seeing the patch.

> We could remove 
> buffering all together in FileSystem level and let the FS 
> implementations to decide how to buffer.

That's already been done, as of HADOOP-928.  FileSystem implementations 
now opt to use ChecksumFileSystem.  The buffer size defaults to 
io.buffer.size, but applications may pass an explicit buffer size to the 
FileSystem.  The FileSystem implementation is free to ignore that hint.

Doug

Mime
View raw message