hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: \r\n problem in LineRecordReader.java
Date Wed, 13 Jun 2007 07:05:50 GMT
Bwolen Yang wrote:
> Here is probably the cause of this bug:
> 
>    public int read(byte b[], int off, int len) throws IOException {
>      // make sure that it ends at a checksum boundary
>      long curPos = getPos();
>      long endPos = len+curPos/bytesPerSum*bytesPerSum;
>      return readBuffer(b, off, (int)(endPos-curPos));
>    }
> 
> Here, the caller calls the function with 127 bytes, and bytesPerSum is 256.

Is this from looking at the code or you actually saw the values like 
this at runtime?

I think 'len' is never supposed to be less than bytesPerChecksum because 
there is a BufferedInputStream between with a buffer size of 
io.buffer.size (default 4096). So this buffer size is supposed to be 
larger than bytesPerChecksum (requirement changes with HADOOP-1450 where 
it uses buffer size equal to bytesPerChecksum).

Raghu.

Mime
View raw message