hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Thu, 26 Aug 2010 18:50:18 GMT
Hey Johannes,

Nice find.  Mind adding the tests you wrote to the patch in
HADOOP-6928 which Owen just filed?

Thanks,
Eli

On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
<jzillmann@googlemail.com> wrote:
> Hi folks,
>
> i'm stumbled into a situation with hadoop-0.20.2 where sorting with a BooleanWritable
doesn't work.
> I wrote some test and it seems to me that the raw-comparator version is the problem...
>
> Test1:
> -------------------------------------------------
>  @Test
>    public void testComparator() throws Exception {
>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>        System.out.println(writableComparator.getClass().getName());
>
>        System.out.println(writableComparator.compare(b1.getData(), 0, b1.size(),
b2.getData(), 0, b2.size()));
>        System.out.println(writableComparator.compare(b2.getData(), 0, b2.size(),
b1.getData(), 0, b1.size()));
>    }
>
>  protected DataOutputBuffer writeWritable(Writable writable) throws IOException {
>        DataOutputBuffer out = new DataOutputBuffer(1024);
>        writable.write(out);
>        out.flush();
>        return out;
>    }
> -------------------------------------------------
> prints-out:
>        org.apache.hadoop.io.BooleanWritable$Comparator
>        0
>        0
> -------------------------------------------------
>
>
> Test2:
> -------------------------------------------------
>  @Test
>    public void testSorterWithRawComparator() throws Exception {
>        sort(new BooleanWritable.Comparator());
>    }
>
>    @Test
>    public void testSorterWithNonRawComparator() throws Exception {
>        sort(new WritableComparator(BooleanWritable.class, true) {});
>    }
>
>    private void sort(RawComparator<BooleanWritable> comparator) throws IOException,
InstantiationException, IllegalAccessException {
>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>        Path inputPath = new Path("/tmp/in");
>        Path outputPath = new Path("/tmp/out");
>        fs.delete(outputPath, true);
>
>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(), inputPath, BooleanWritable.class,
Text.class);
>        writer.append(new BooleanWritable(true), new Text(""));
>        writer.append(new BooleanWritable(false), new Text(""));
>        writer.append(new BooleanWritable(true), new Text(""));
>        writer.append(new BooleanWritable(true), new Text(""));
>        writer.append(new BooleanWritable(false), new Text(""));
>        writer.close();
>
>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class, Text.class,
fs.getConf());
>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>
>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>        Writable key = (Writable) reader.getKeyClass().newInstance();
>        Writable value = (Writable) reader.getValueClass().newInstance();
>
>        System.out.println("--------------------");
>        while (reader.next(key, value)) {
>            System.out.println(key);
>        }
>        reader.close();
>    }
> -------------------------------------------------
> prints-out:
>        org.apache.hadoop.io.BooleanWritable$Comparator: true false true true false
>        datameer.dap.common.experimental.BooleanWritableTest$1: false false true true
true
> -------------------------------------------------
>
> Or am i missing something ?
> best regards
> Johannes

Mime
View raw message