hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: Many Checksum Errors
Date Wed, 16 May 2007 19:31:35 GMT
Doug Cutting wrote:
> Raghu Angadi wrote:
>> But this will not fix the same problem with block-level checksums. 
>> Pretty soon, HDFS will not use ChecksumFileSystem at all.
> I'd hope that block-level checksums do not replicate logic from 
> ChecksumFileSystem.  Rather they should probably share substantial 
> portions of their checksumming input and output stream implementations, 
> no?  So it could fix the same problem for block-level checksums, and 
> should if possible.

Nope, DFSClient that implements client side of block-levels checksums 
does not replicate or reuse any code from ChecksumFileSystem.

>> Ideally we should let the implementations decide how to buffer.
> I'm not sure what you mean by this.  The buffer size is a parameter to 
> FileSystem's open() and create() methods. 

All FSOutputStreams (including DFS) go through a BufferedOutputStream. 
We can not give buffersize of 0. But may be DistributedFileSystem can 
always provide bufferSize of 1. I will see we can easily support 
explicit option not to buffer to FSOutputStream.

> Whether checksums require 
> another level of buffering is a separate issue.

Yes. In my patch, DFS uses FSOutputStream with default buffer so DFS is 
affected by the same issue, in a different place.

> Is it efficient to 
> invoke the CRC32 code as each byte is written, or is it faster to run it 
> in 512-byte or larger batches?

I think CPU cost wise, it is twice as fast to CRC32 larger buffers (> 
512) than to CRC32 small buffers. But I don't think its a very 
noticeable overhead. Do we expect users do many small writes?

Some measurements I did some time back:

Total size  read size  Total Overhead  Overhead/MB    MB/Overhead
   128 MB      64	1.3  sec       10.1  msec/MB  100 MB/sec
   128	     128	0.98		7.65	      130
   128	     256	0.80		6.24	      160
   128	     512	0.71		5.52	      180
   128	    1024	0.68		5.25	      190
   128	   10240	0.65		5.12	      195

> Doug

View raw message