Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=DeuHqMrMhUWKhx5F6YZLDGM8Kg+ipBg6ORsGncVtAZ4oyn6Ca83mpQKaucChWYDY
Message-ID: <466F974E.90403@yahoo-inc.com>
Date: Wed, 13 Jun 2007 00:05:50 -0700
From: Raghu Angadi <rangadi@yahoo-inc.com>
User-Agent: Thunderbird 1.5.0.12 (Windows/20070509)
MIME-Version: 1.0
To: hadoop-dev@lucene.apache.org
Subject: Re: \r\n problem in LineRecordReader.java
References: <e8ce6bc30706121035q18723d3v77721087c850e2fc@mail.gmail.com>
 <e8ce6bc30706122228w17cedb6as83b2c93ae0f82d65@mail.gmail.com>
In-Reply-To: <e8ce6bc30706122228w17cedb6as83b2c93ae0f82d65@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Bwolen Yang wrote:
> Here is probably the cause of this bug:
> 
>    public int read(byte b[], int off, int len) throws IOException {
>      // make sure that it ends at a checksum boundary
>      long curPos = getPos();
>      long endPos = len+curPos/bytesPerSum*bytesPerSum;
>      return readBuffer(b, off, (int)(endPos-curPos));
>    }
> 
> Here, the caller calls the function with 127 bytes, and bytesPerSum is 256.

Is this from looking at the code or you actually saw the values like 
this at runtime?

I think 'len' is never supposed to be less than bytesPerChecksum because 
there is a BufferedInputStream between with a buffer size of 
io.buffer.size (default 4096). So this buffer size is supposed to be 
larger than bytesPerChecksum (requirement changes with HADOOP-1450 where 
it uses buffer size equal to bytesPerChecksum).

Raghu.