hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Zillmann <jzillm...@googlemail.com>
Subject Re: RawComparator of BooleanWritable broken ?
Date Fri, 27 Aug 2010 07:07:17 GMT
Hey Eli,

i would do, but i need some help on setting up the project in the correct version! I had problem
figuring that out and i posted this 2 days ago to the users list already (but have not received
any answer yet):
----------------------
 	svn/git revisions for 0.20.2
Hey folks,
can somebody tell me how to get the source versions from git/svn for hadoop-hdfs and hadoop-mapreduce
?
In hadoop-common there are branches and tags for the release. But how to get the corresponding
version of the other 2 projects ?
----------------------

best regards
Johannes


On Aug 27, 2010, at 12:15 AM, Eli Collins wrote:

> Hey Johannes,
> 
> Unfortunately I can't cut-copy-paste your code from the email as in
> order for them to be included by the project they have to be granted
> to the ASF (there's a button when you attach a patch in jira that does
> this) for licensing reasons.
> 
> Could you generate a patch that modifies TestComparators.java (or adds
> a new test class) that includes the tests you wrote?  Info on how to
> do this:  http://wiki.apache.org/hadoop/HowToContribute
> 
> Thanks,
> Eli
> 
> 
> On Thu, Aug 26, 2010 at 3:02 PM, Johannes Zillmann
> <jzillmann@googlemail.com> wrote:
>> Hey Eli,
>> 
>> no, please add it!
>> @Owen, Thanks !
>> 
>> Johannes
>> 
>> On Aug 26, 2010, at 8:50 PM, Eli Collins wrote:
>> 
>>> Hey Johannes,
>>> 
>>> Nice find.  Mind adding the tests you wrote to the patch in
>>> HADOOP-6928 which Owen just filed?
>>> 
>>> Thanks,
>>> Eli
>>> 
>>> On Thu, Aug 26, 2010 at 4:57 AM, Johannes Zillmann
>>> <jzillmann@googlemail.com> wrote:
>>>> Hi folks,
>>>> 
>>>> i'm stumbled into a situation with hadoop-0.20.2 where sorting with a BooleanWritable
doesn't work.
>>>> I wrote some test and it seems to me that the raw-comparator version is the
problem...
>>>> 
>>>> Test1:
>>>> -------------------------------------------------
>>>>  @Test
>>>>    public void testComparator() throws Exception {
>>>>        DataOutputBuffer b1 = writeWritable(new BooleanWritable(true));
>>>>        DataOutputBuffer b2 = writeWritable(new BooleanWritable(false));
>>>>        WritableComparator writableComparator = WritableComparator.get(BooleanWritable.class);
>>>>        System.out.println(writableComparator.getClass().getName());
>>>> 
>>>>        System.out.println(writableComparator.compare(b1.getData(), 0, b1.size(),
b2.getData(), 0, b2.size()));
>>>>        System.out.println(writableComparator.compare(b2.getData(), 0, b2.size(),
b1.getData(), 0, b1.size()));
>>>>    }
>>>> 
>>>>  protected DataOutputBuffer writeWritable(Writable writable) throws IOException
{
>>>>        DataOutputBuffer out = new DataOutputBuffer(1024);
>>>>        writable.write(out);
>>>>        out.flush();
>>>>        return out;
>>>>    }
>>>> -------------------------------------------------
>>>> prints-out:
>>>>        org.apache.hadoop.io.BooleanWritable$Comparator
>>>>        0
>>>>        0
>>>> -------------------------------------------------
>>>> 
>>>> 
>>>> Test2:
>>>> -------------------------------------------------
>>>>  @Test
>>>>    public void testSorterWithRawComparator() throws Exception {
>>>>        sort(new BooleanWritable.Comparator());
>>>>    }
>>>> 
>>>>    @Test
>>>>    public void testSorterWithNonRawComparator() throws Exception {
>>>>        sort(new WritableComparator(BooleanWritable.class, true) {});
>>>>    }
>>>> 
>>>>    private void sort(RawComparator<BooleanWritable> comparator) throws
IOException, InstantiationException, IllegalAccessException {
>>>>        LocalFileSystem fs = FileSystem.getLocal(new Configuration());
>>>>        Path inputPath = new Path("/tmp/in");
>>>>        Path outputPath = new Path("/tmp/out");
>>>>        fs.delete(outputPath, true);
>>>> 
>>>>        Writer writer = SequenceFile.createWriter(fs, fs.getConf(), inputPath,
BooleanWritable.class, Text.class);
>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>        writer.append(new BooleanWritable(true), new Text(""));
>>>>        writer.append(new BooleanWritable(false), new Text(""));
>>>>        writer.close();
>>>> 
>>>>        Sorter sorter = new Sorter(fs, comparator, BooleanWritable.class,
Text.class, fs.getConf());
>>>>        sorter.sort(new Path[] { inputPath }, outputPath, true);
>>>> 
>>>>        Reader reader = new SequenceFile.Reader(fs, outputPath, fs.getConf());
>>>>        Writable key = (Writable) reader.getKeyClass().newInstance();
>>>>        Writable value = (Writable) reader.getValueClass().newInstance();
>>>> 
>>>>        System.out.println("--------------------");
>>>>        while (reader.next(key, value)) {
>>>>            System.out.println(key);
>>>>        }
>>>>        reader.close();
>>>>    }
>>>> -------------------------------------------------
>>>> prints-out:
>>>>        org.apache.hadoop.io.BooleanWritable$Comparator: true false true true
false
>>>>        datameer.dap.common.experimental.BooleanWritableTest$1: false false
true true true
>>>> -------------------------------------------------
>>>> 
>>>> Or am i missing something ?
>>>> best regards
>>>> Johannes
>> 
>> 


Mime
View raw message