hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <owen.omal...@gmail.com>
Subject Re: Anyway to sort "keys" before Reduce function in Hadoop ?
Date Wed, 17 Jun 2009 23:12:43 GMT
On Wed, Jun 17, 2009 at 12:26 PM, Chuck Lam <chuck.lam@gmail.com> wrote:

> an alternative is to create a new WritableComparator and then set it
> in the JobConf object with the method setOutputKeyComparatorClass().
> You can use IntWritable.Comparator as a start.


The important part of that is to define a RawComparator for your key class
and call JobConf.setOutputKeyComparatorClass with it.So if you wanted to
invert the default sort order of IntWritable keys, you could:

public class InvertedIntWritableComparator extends IntWritable.Comparator {

    public int compare(byte[] b1, int s1, int l1,
                       byte[] b2, int s2, int l2) {
      return -1 * super.compare(b1,s1,l1,b2,s2,l2);
    }
}

then

job.setOutputKeyComparatorClass(InvertedIntWritableComparator.class);

-- Owen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message