hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Zillmann <jzillm...@googlemail.com>
Subject RawComparator of BooleanWritable broken ?
Date Thu, 26 Aug 2010 11:57:51 GMT
Hi folks,

i'm stumbled into a situation with hadoop-0.20.2 where sorting with a BooleanWritable doesn't
work.
I wrote some test and it seems to me that the raw-comparator version is the problem...

Test1:
-------------------------------------------------
 @Test
    public void testComparator() throws Exception {
        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
        System.out.println(writableComparator.getClass().getName());

        System.out.println(writableComparator.compare(b1.getData(), 0, b1.size(), b2.getData(),
0, b2.size()));
        System.out.println(writableComparator.compare(b2.getData(), 0, b2.size(), b1.getData(),
0, b1.size()));
    }

 protected DataOutputBuffer writeWritable(Writable writable) throws IOException {
        DataOutputBuffer out = new DataOutputBuffer(1024);
        writable.write(out);
        out.flush();
        return out;
    }
-------------------------------------------------
prints-out:	
	org.apache.hadoop.io.BooleanWritable$Comparator
	0
	0
-------------------------------------------------


Test2:
-------------------------------------------------
 @Test
    public void testSorterWithRawComparator() throws Exception {
        sort(new BooleanWritable.Comparator());
    }

    @Test
    public void testSorterWithNonRawComparator() throws Exception {
        sort(new WritableComparator(BooleanWritable.class, true) {});
    }

    private void sort(RawComparator<BooleanWritable> comparator) throws IOException,
InstantiationException, IllegalAccessException {
        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
        Path inputPath = new Path("/tmp/in");
        Path outputPath = new Path("/tmp/out");
        fs.delete(outputPath, true);

        Writer writer = SequenceFile.createWriter(fs, fs.getConf(), inputPath, BooleanWritable.class,
Text.class);
        writer.append(new BooleanWritable(true), new Text(""));
        writer.append(new BooleanWritable(false), new Text(""));
        writer.append(new BooleanWritable(true), new Text(""));
        writer.append(new BooleanWritable(true), new Text(""));
        writer.append(new BooleanWritable(false), new Text(""));
        writer.close();

        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class, Text.class, fs.getConf());
        sorter.sort(new Path[] { inputPath }, outputPath, true);

        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
        Writable key = (Writable) reader.getKeyClass().newInstance();
        Writable value = (Writable) reader.getValueClass().newInstance();

        System.out.println("--------------------");
        while (reader.next(key, value)) {
            System.out.println(key);
        }
        reader.close();
    }
-------------------------------------------------
prints-out:	
	org.apache.hadoop.io.BooleanWritable$Comparator: true false true true false 
	datameer.dap.common.experimental.BooleanWritableTest$1: false false true true true
-------------------------------------------------

Or am i missing something ?
best regards
Johannes
Mime
View raw message