hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Fri, 27 Aug 2010 16:29:49 GMT
Hey Johannes,

For branch 20 all the common/hdfs/mapreduce code lives in branch-0.20
on hadoop-common.git  (after 20 the code was split into three
repositories).

Wrt to the jira I think we need:
1. A patch to trunk that adds a new test class
src/test/org/apache/hadoop/io/TestBooleanWritable.java that has your
testComparator test  (trunk already has the fix)
2. Modify h-6928.patch (which is for branch 20) to include the above test

Thanks,
Eli

On Fri, Aug 27, 2010 at 12:07 AM, Johannes Zillmann
<jzillmann@googlemail.com> wrote:
> Hey Eli,
>
> i would do, but i need some help on setting up the project in the correct version! I
had problem figuring that out and i posted this 2 days ago to the users list already (but
have not received any answer yet):
> ----------------------
>        svn/git revisions for 0.20.2
> Hey folks,
> can somebody tell me how to get the source versions from git/svn for hadoop-hdfs and
hadoop-mapreduce ?
> In hadoop-common there are branches and tags for the release. But how to get the corresponding
version of the other 2 projects ?
> ----------------------
>
> best regards
> Johannes
>
>
> On Aug 27, 2010, at 12:15 AM, Eli Collins wrote:
>
>> Hey Johannes,
>>
>> Unfortunately I can't cut-copy-paste your code from the email as in
>> order for them to be included by the project they have to be granted
>> to the ASF (there's a button when you attach a patch in jira that does
>> this) for licensing reasons.
>>
>> Could you generate a patch that modifies TestComparators.java (or adds
>> a new test class) that includes the tests you wrote?  Info on how to
>> do this:  http://wiki.apache.org/hadoop/HowToContribute
>>
>> Thanks,
>> Eli
>>
>>
>> On Thu, Aug 26, 2010 at 3:02 PM, Johannes Zillmann
>> <jzillmann@googlemail.com> wrote:
>>> Hey Eli,
>>>
>>> no, please add it!
>>> @Owen, Thanks !
>>>
>>> Johannes
>>>
>>> On Aug 26, 2010, at 8:50 PM, Eli Collins wrote:
>>>
>>>> Hey Johannes,
>>>>
>>>> Nice find.  Mind adding the tests you wrote to the patch in
>>>> HADOOP-6928 which Owen just filed?
>>>>
>>>> Thanks,
>>>> Eli
>>>>
>>>> On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
>>>> <jzillmann@googlemail.com> wrote:
>>>>> Hi folks,
>>>>>
>>>>> i'm stumbled into a situation with hadoop-0.20.2 where sorting with a
BooleanWritable doesn't work.
>>>>> I wrote some test and it seems to me that the raw-comparator version
is the problem...
>>>>>
>>>>> Test1:
>>>>> -------------------------------------------------
>>>>>  @Test
>>>>>    public void testComparator() throws Exception {
>>>>>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>>>>>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>>>>>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>>>>>        System.out.println(writableComparator.getClass().getName());
>>>>>
>>>>>        System.out.println(writableComparator.compare(b1.getData(),
0, b1.size(), b2.getData(), 0, b2.size()));
>>>>>        System.out.println(writableComparator.compare(b2.getData(),
0, b2.size(), b1.getData(), 0, b1.size()));
>>>>>    }
>>>>>
>>>>>  protected DataOutputBuffer writeWritable(Writable writable) throws
IOException {
>>>>>        DataOutputBuffer out = new DataOutputBuffer(1024);
>>>>>        writable.write(out);
>>>>>        out.flush();
>>>>>        return out;
>>>>>    }
>>>>> -------------------------------------------------
>>>>> prints-out:
>>>>>        org.apache.hadoop.io.BooleanWritable$Comparator
>>>>>        0
>>>>>        0
>>>>> -------------------------------------------------
>>>>>
>>>>>
>>>>> Test2:
>>>>> -------------------------------------------------
>>>>>  @Test
>>>>>    public void testSorterWithRawComparator() throws Exception {
>>>>>        sort(new BooleanWritable.Comparator());
>>>>>    }
>>>>>
>>>>>    @Test
>>>>>    public void testSorterWithNonRawComparator() throws Exception {
>>>>>        sort(new WritableComparator(BooleanWritable.class, true) {});
>>>>>    }
>>>>>
>>>>>    private void sort(RawComparator<BooleanWritable> comparator)
throws IOException, InstantiationException, IllegalAccessException {
>>>>>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>>>>>        Path inputPath = new Path("/tmp/in");
>>>>>        Path outputPath = new Path("/tmp/out");
>>>>>        fs.delete(outputPath, true);
>>>>>
>>>>>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(),
inputPath, BooleanWritable.class, Text.class);
>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>>        writer.close();
>>>>>
>>>>>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class,
Text.class, fs.getConf());
>>>>>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>>>>>
>>>>>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>>>>>        Writable key = (Writable) reader.getKeyClass().newInstance();
>>>>>        Writable value = (Writable) reader.getValueClass().newInstance();
>>>>>
>>>>>        System.out.println("--------------------");
>>>>>        while (reader.next(key, value)) {
>>>>>            System.out.println(key);
>>>>>        }
>>>>>        reader.close();
>>>>>    }
>>>>> -------------------------------------------------
>>>>> prints-out:
>>>>>        org.apache.hadoop.io.BooleanWritable$Comparator: true false
true true false
>>>>>        datameer.dap.common.experimental.BooleanWritableTest$1: false
false true true true
>>>>> -------------------------------------------------
>>>>>
>>>>> Or am i missing something ?
>>>>> best regards
>>>>> Johannes
>>>
>>>
>
>

Mime
View raw message