hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahad Rana (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6834) TFile.append compares initial key against null lastKey
Date Tue, 22 Jun 2010 21:40:57 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881372#action_12881372

Ahad Rana commented on HADOOP-6834:

Hi Hong,

Sure, that would be great. What are the requirements these days ? Which version of hadoop
should the patch target ? 


> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically
NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append.
The underlying RawComparator interface used for comparisons does not stipulate the proper
behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator
implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates
the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's
comparator, which ignores length and thus produces incorrect results. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message