hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Thu, 26 Aug 2010 22:15:47 GMT
Hey Johannes,

Unfortunately I can't cut-copy-paste your code from the email as in
order for them to be included by the project they have to be granted
to the ASF (there's a button when you attach a patch in jira that does
this) for licensing reasons.

Could you generate a patch that modifies TestComparators.java (or adds
a new test class) that includes the tests you wrote?  Info on how to
do this:  http://wiki.apache.org/hadoop/HowToContribute

Thanks,
Eli


On Thu, Aug 26, 2010 at 3:02 PM, Johannes Zillmann
<jzillmann@googlemail.com> wrote:
> Hey Eli,
>
> no, please add it!
> @Owen, Thanks !
>
> Johannes
>
> On Aug 26, 2010, at 8:50 PM, Eli Collins wrote:
>
>> Hey Johannes,
>>
>> Nice find.  Mind adding the tests you wrote to the patch in
>> HADOOP-6928 which Owen just filed?
>>
>> Thanks,
>> Eli
>>
>> On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
>> <jzillmann@googlemail.com> wrote:
>>> Hi folks,
>>>
>>> i'm stumbled into a situation with hadoop-0.20.2 where sorting with a BooleanWritable
doesn't work.
>>> I wrote some test and it seems to me that the raw-comparator version is the problem...
>>>
>>> Test1:
>>> -------------------------------------------------
>>>  @Test
>>>    public void testComparator() throws Exception {
>>>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>>>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>>>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>>>        System.out.println(writableComparator.getClass().getName());
>>>
>>>        System.out.println(writableComparator.compare(b1.getData(), 0, b1.size(),
b2.getData(), 0, b2.size()));
>>>        System.out.println(writableComparator.compare(b2.getData(), 0, b2.size(),
b1.getData(), 0, b1.size()));
>>>    }
>>>
>>>  protected DataOutputBuffer writeWritable(Writable writable) throws IOException
{
>>>        DataOutputBuffer out = new DataOutputBuffer(1024);
>>>        writable.write(out);
>>>        out.flush();
>>>        return out;
>>>    }
>>> -------------------------------------------------
>>> prints-out:
>>>        org.apache.hadoop.io.BooleanWritable$Comparator
>>>        0
>>>        0
>>> -------------------------------------------------
>>>
>>>
>>> Test2:
>>> -------------------------------------------------
>>>  @Test
>>>    public void testSorterWithRawComparator() throws Exception {
>>>        sort(new BooleanWritable.Comparator());
>>>    }
>>>
>>>    @Test
>>>    public void testSorterWithNonRawComparator() throws Exception {
>>>        sort(new WritableComparator(BooleanWritable.class, true) {});
>>>    }
>>>
>>>    private void sort(RawComparator<BooleanWritable> comparator) throws
IOException, InstantiationException, IllegalAccessException {
>>>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>>>        Path inputPath = new Path("/tmp/in");
>>>        Path outputPath = new Path("/tmp/out");
>>>        fs.delete(outputPath, true);
>>>
>>>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(), inputPath,
BooleanWritable.class, Text.class);
>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>        writer.close();
>>>
>>>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class,
Text.class, fs.getConf());
>>>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>>>
>>>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>>>        Writable key = (Writable) reader.getKeyClass().newInstance();
>>>        Writable value = (Writable) reader.getValueClass().newInstance();
>>>
>>>        System.out.println("--------------------");
>>>        while (reader.next(key, value)) {
>>>            System.out.println(key);
>>>        }
>>>        reader.close();
>>>    }
>>> -------------------------------------------------
>>> prints-out:
>>>        org.apache.hadoop.io.BooleanWritable$Comparator: true false true true
false
>>>        datameer.dap.common.experimental.BooleanWritableTest$1: false false
true true true
>>> -------------------------------------------------
>>>
>>> Or am i missing something ?
>>> best regards
>>> Johannes
>
>

Mime
View raw message