hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Zillmann <jzillm...@googlemail.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Thu, 26 Aug 2010 22:02:41 GMT
Hey Eli,

no, please add it!
@Owen, Thanks !

Johannes

On Aug 26, 2010, at 8:50 PM, Eli Collins wrote:

> Hey Johannes,
> 
> Nice find.  Mind adding the tests you wrote to the patch in
> HADOOP-6928 which Owen just filed?
> 
> Thanks,
> Eli
> 
> On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
> <jzillmann@googlemail.com> wrote:
>> Hi folks,
>> 
>> i'm stumbled into a situation with hadoop-0.20.2 where sorting with a BooleanWritable
doesn't work.
>> I wrote some test and it seems to me that the raw-comparator version is the problem...
>> 
>> Test1:
>> -------------------------------------------------
>>  @Test
>>    public void testComparator() throws Exception {
>>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>>        System.out.println(writableComparator.getClass().getName());
>> 
>>        System.out.println(writableComparator.compare(b1.getData(), 0, b1.size(),
b2.getData(), 0, b2.size()));
>>        System.out.println(writableComparator.compare(b2.getData(), 0, b2.size(),
b1.getData(), 0, b1.size()));
>>    }
>> 
>>  protected DataOutputBuffer writeWritable(Writable writable) throws IOException {
>>        DataOutputBuffer out = new DataOutputBuffer(1024);
>>        writable.write(out);
>>        out.flush();
>>        return out;
>>    }
>> -------------------------------------------------
>> prints-out:
>>        org.apache.hadoop.io.BooleanWritable$Comparator
>>        0
>>        0
>> -------------------------------------------------
>> 
>> 
>> Test2:
>> -------------------------------------------------
>>  @Test
>>    public void testSorterWithRawComparator() throws Exception {
>>        sort(new BooleanWritable.Comparator());
>>    }
>> 
>>    @Test
>>    public void testSorterWithNonRawComparator() throws Exception {
>>        sort(new WritableComparator(BooleanWritable.class, true) {});
>>    }
>> 
>>    private void sort(RawComparator<BooleanWritable> comparator) throws IOException,
InstantiationException, IllegalAccessException {
>>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>>        Path inputPath = new Path("/tmp/in");
>>        Path outputPath = new Path("/tmp/out");
>>        fs.delete(outputPath, true);
>> 
>>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(), inputPath, BooleanWritable.class,
Text.class);
>>        writer.append(new BooleanWritable(true), new Text(""));
>>        writer.append(new BooleanWritable(false), new Text(""));
>>        writer.append(new BooleanWritable(true), new Text(""));
>>        writer.append(new BooleanWritable(true), new Text(""));
>>        writer.append(new BooleanWritable(false), new Text(""));
>>        writer.close();
>> 
>>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class, Text.class,
fs.getConf());
>>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>> 
>>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>>        Writable key = (Writable) reader.getKeyClass().newInstance();
>>        Writable value = (Writable) reader.getValueClass().newInstance();
>> 
>>        System.out.println("--------------------");
>>        while (reader.next(key, value)) {
>>            System.out.println(key);
>>        }
>>        reader.close();
>>    }
>> -------------------------------------------------
>> prints-out:
>>        org.apache.hadoop.io.BooleanWritable$Comparator: true false true true false
>>        datameer.dap.common.experimental.BooleanWritableTest$1: false false true true
true
>> -------------------------------------------------
>> 
>> Or am i missing something ?
>> best regards
>> Johannes


Mime
View raw message