Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of imaximchuk@masterhost.ru
 designates 83.222.24.118 as permitted sender)
Message-ID: <48B81E64.1020201@masterhost.ru>
Date: Fri, 29 Aug 2008 20:05:56 +0400
From: Igor Maximchuk <imaximchuk@masterhost.ru>
User-Agent: Thunderbird 2.0.0.16 (Windows/20080708)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Some comparators defined via .JobConf.setOutputKeyComparatorClass
 no longer work in 0.18.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello,

I use a custom comparators that  subclass  WritableComparator and 
overrides int  compare(WritableComparable lh, WritableComparable rh)

for example:

public class ProfoutComparator extends WritableComparator {
    public ProfoutComparator() {
        super(ProfoutMapKey.class);
    }

    public int compare(WritableComparable lh, WritableComparable rh) {
        ProfoutMapKey lk = (ProfoutMapKey) lh;
        ProfoutMapKey rk = (ProfoutMapKey) rh;
        if (lk.getVisitorId() > rk.getVisitorId())return 1;
        if (lk.getVisitorId() < rk.getVisitorId())return -1;
        if (lk.getInterestValue() < rk.getInterestValue())return 1;
        if (lk.getInterestValue() > rk.getInterestValue())return -1;
        return 0;
    }
}

I need different comparators of MyKey for different mapreduce jobs. so I 
cannot simply override compareTo method in MyKey and cannot register 
custom comparator for MyKey because

after upgrading to 0.18 I start receving NullPointerExceptions  when 
running tasks that define comparator via 
.JobConf.setOutputKeyComparatorClass

08/08/29 19:49:20 INFO mapred.TaskInProgress: Error from 
attempt_200808291949_0001_m_000000_1: java.lang.NullPointerException
    at 
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:487)
    at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
    at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
    at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:750)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
    at 
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)


looks like this is because "buffer" variable is not initialised, since 
default constructor for  WritableComparator no longer initialises it and 
there is no way to make it do so. See patches in  
https://issues.apache.org/jira/browse/HADOOP-3665.

Is it possible to change signature of   WritableComparator(Class 
keyClass, boolean createInstances) from private to protected so it can 
be called from subclass or make another way to tell WritableComparator 
to initialize fully?