hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sanda <john.sa...@gmail.com>
Subject changing key comparator used with chain mappers
Date Wed, 09 Mar 2011 21:32:17 GMT
If I want to change how keys are sorted prior to the reduce, my
understanding is that I can do this with
JobConf.setOutputKeyComparatorClass(). I am trying to implement a job using
ChainReducer such that I have,

Map 1 | Reduce | Map 2

and I want to use my own comparator for Map 2 so that all keys will sorted
according to my comparator at the end of the job. I tried the following,

ChainReducer.setReducer(

    job,

    Reducer.class,

    Text.class,

    Text.class,

    Text.class,

    Text.class,

    false,

    reducerConf);


JobConf map2Conf = new JobConf(false);

sortConf.setOutputKeyComparatorClass(MyComparator.class);

ChainReducer.addMapper(

    job,

    IdentityMapper.class,

    Text.class,

    Text.class,

    Text.class,

    Text.class,

    true,
    map2ConfConf);

This didn't seem to work. How can I change the sort order of the keys coming
out of Map 2?

Thanks

- John

Mime
View raw message