hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahad Rana (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6834) TFile.append compares initial key against null lastKey
Date Tue, 22 Jun 2010 21:10:57 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881362#action_12881362

Ahad Rana commented on HADOOP-6834:

Hi Hong,

Yes I would be happy to provide a patch. As far as triggering the failure, if you append a
LongWritable with a negative value as the first key value and use the LongWritable's default
Comparator, this will trigger the bug. I would be happy to provide a code snippet if required.
In the case of LongWritable's comparator, it interprets the uninitialized zero bytes of the
lastKey as a zero (since it does not check length), and thus this causes the out-of-order
insertion exception. Either LongWritable's comparator is implemented improperly, or the contract
implied by RawComparator is not clear on what happens in the case where you compare a non-zero
byte array to a zero byte array. Probably both LongWritable's RawComparator should be fixed,
but either way, passing in a uninitialized lastKey is going to cause problems for someone
else down the road, so probably best to fix this as well ?     


> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically
NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append.
The underlying RawComparator interface used for comparisons does not stipulate the proper
behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator
implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates
the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's
comparator, which ignores length and thus produces incorrect results. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message