hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Zillmann <jzillm...@googlemail.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Sat, 28 Aug 2010 17:57:18 GMT
Wow, 

what effort for such a tiny piece of code!
I did as you advised, thanks for your guidance!

Johannes

On Aug 27, 2010, at 6:29 PM, Eli Collins wrote:

> Hey Johannes,
> 
> For branch 20 all the common/hdfs/mapreduce code lives in branch-0.20
> on hadoop-common.git  (after 20 the code was split into three
> repositories).
> 
> Wrt to the jira I think we need:
> 1. A patch to trunk that adds a new test class
> src/test/org/apache/hadoop/io/TestBooleanWritable.java that has your
> testComparator test  (trunk already has the fix)
> 2. Modify h-6928.patch (which is for branch 20) to include the above test
> 
> Thanks,
> Eli
> 
> On Fri, Aug 27, 2010 at 12:07 AM, Johannes Zillmann
> <jzillmann@googlemail.com> wrote:
>> Hey Eli,
>> 
>> i would do, but i need some help on setting up the project in the correct version!
I had problem figuring that out and i posted this 2 days ago to the users list already (but
have not received any answer yet):
>> ----------------------
>>        svn/git revisions for 0.20.2
>> Hey folks,
>> can somebody tell me how to get the source versions from git/svn for hadoop-hdfs
and hadoop-mapreduce ?
>> In hadoop-common there are branches and tags for the release. But how to get the
corresponding version of the other 2 projects ?
>> ----------------------
>> 
>> best regards
>> Johannes
>> 
>> 
>> On Aug 27, 2010, at 12:15 AM, Eli Collins wrote:
>> 
>>> Hey Johannes,
>>> 
>>> Unfortunately I can't cut-copy-paste your code from the email as in
>>> order for them to be included by the project they have to be granted
>>> to the ASF (there's a button when you attach a patch in jira that does
>>> this) for licensing reasons.
>>> 
>>> Could you generate a patch that modifies TestComparators.java (or adds
>>> a new test class) that includes the tests you wrote?  Info on how to
>>> do this:  http://wiki.apache.org/hadoop/HowToContribute
>>> 
>>> Thanks,
>>> Eli
>>> 
>>> 
>>> On Thu, Aug 26, 2010 at 3:02 PM, Johannes Zillmann
>>> <jzillmann@googlemail.com> wrote:
>>>> Hey Eli,
>>>> 
>>>> no, please add it!
>>>> @Owen, Thanks !
>>>> 
>>>> Johannes
>>>> 
>>>> On Aug 26, 2010, at 8:50 PM, Eli Collins wrote:
>>>> 
>>>>> Hey Johannes,
>>>>> 
>>>>> Nice find.  Mind adding the tests you wrote to the patch in
>>>>> HADOOP-6928 which Owen just filed?
>>>>> 
>>>>> Thanks,
>>>>> Eli
>>>>> 
>>>>> On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
>>>>> <jzillmann@googlemail.com> wrote:
>>>>>> Hi folks,
>>>>>> 
>>>>>> i'm stumbled into a situation with hadoop-0.20.2 where sorting with
a BooleanWritable doesn't work.
>>>>>> I wrote some test and it seems to me that the raw-comparator version
is the problem...
>>>>>> 
>>>>>> Test1:
>>>>>> -------------------------------------------------
>>>>>>  @Test
>>>>>>    public void testComparator() throws Exception {
>>>>>>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>>>>>>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>>>>>>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>>>>>>        System.out.println(writableComparator.getClass().getName());
>>>>>> 
>>>>>>        System.out.println(writableComparator.compare(b1.getData(),
0, b1.size(), b2.getData(), 0, b2.size()));
>>>>>>        System.out.println(writableComparator.compare(b2.getData(),
0, b2.size(), b1.getData(), 0, b1.size()));
>>>>>>    }
>>>>>> 
>>>>>>  protected DataOutputBuffer writeWritable(Writable writable) throws
IOException {
>>>>>>        DataOutputBuffer out = new DataOutputBuffer(1024);
>>>>>>        writable.write(out);
>>>>>>        out.flush();
>>>>>>        return out;
>>>>>>    }
>>>>>> -------------------------------------------------
>>>>>> prints-out:
>>>>>>        org.apache.hadoop.io.BooleanWritable$Comparator
>>>>>>        0
>>>>>>        0
>>>>>> -------------------------------------------------
>>>>>> 
>>>>>> 
>>>>>> Test2:
>>>>>> -------------------------------------------------
>>>>>>  @Test
>>>>>>    public void testSorterWithRawComparator() throws Exception {
>>>>>>        sort(new BooleanWritable.Comparator());
>>>>>>    }
>>>>>> 
>>>>>>    @Test
>>>>>>    public void testSorterWithNonRawComparator() throws Exception
{
>>>>>>        sort(new WritableComparator(BooleanWritable.class, true) {});
>>>>>>    }
>>>>>> 
>>>>>>    private void sort(RawComparator<BooleanWritable> comparator)
throws IOException, InstantiationException, IllegalAccessException {
>>>>>>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>>>>>>        Path inputPath = new Path("/tmp/in");
>>>>>>        Path outputPath = new Path("/tmp/out");
>>>>>>        fs.delete(outputPath, true);
>>>>>> 
>>>>>>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(),
inputPath, BooleanWritable.class, Text.class);
>>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>>>        writer.close();
>>>>>> 
>>>>>>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class,
Text.class, fs.getConf());
>>>>>>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>>>>>> 
>>>>>>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>>>>>>        Writable key = (Writable) reader.getKeyClass().newInstance();
>>>>>>        Writable value = (Writable) reader.getValueClass().newInstance();
>>>>>> 
>>>>>>        System.out.println("--------------------");
>>>>>>        while (reader.next(key, value)) {
>>>>>>            System.out.println(key);
>>>>>>        }
>>>>>>        reader.close();
>>>>>>    }
>>>>>> -------------------------------------------------
>>>>>> prints-out:
>>>>>>        org.apache.hadoop.io.BooleanWritable$Comparator: true false
true true false
>>>>>>        datameer.dap.common.experimental.BooleanWritableTest$1: false
false true true true
>>>>>> -------------------------------------------------
>>>>>> 
>>>>>> Or am i missing something ?
>>>>>> best regards
>>>>>> Johannes
>>>> 
>>>> 
>> 
>> 


Mime
View raw message