hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanley Xu <wenhao...@gmail.com>
Subject Re: Could we use different output Format for the Mapper and Combiner?
Date Thu, 17 Feb 2011 07:01:41 GMT
Hi Alain,

I thought Hash is correct for I found that from the old mails.

http://search-hadoop.com/m/eSd3VxxvkC1/combiner+input+output+format&subj=What+s+a+valid+combiner+

And per my test, the combiner probably could not has the different output
type with the mapper, at least until v 0.20.3.

Thanks for both of you.

Best wishes,
Stanley Xu



On Wed, Feb 16, 2011 at 9:38 PM, MONTMORY Alain <
alain.montmory@thalesgroup.com> wrote:

>  Hi,
>
>
>
> I think you could use different type for mapper and combiner, they are not
> linked together but suppose :
>
>
>
> maper < KeyTypeA, ValuetypeB>
>
> reducer < KeyTypeC, ValuetypeD>
>
>
>
> in your *mapper* you have to emit :
>
> *public* *void* map(KeyTypeA, ValuetypeB)
>
> {
>
>                 ….**
>
> * *
>
> *      *context.write(*KeyTypeC, ValuetypeD*);
>
>
>
> }
>
>
>
> hopes this help!
>
>
>
> regards
>
>
> Alain
>
>
>
> [@@THALES GROUP RESTRICTED@@]
>
>
>
> *De :* Stanley Xu [mailto:wenhao.xu@gmail.com]
> *Envoyé :* mercredi 16 février 2011 12:02
> *À :* mapreduce-user@hadoop.apache.org
> *Objet :* Could we use different output Format for the Mapper and
> Combiner?
>
>
>
> Dear all,
>
>
>
> I am writing a map-reduce job today. Which I hope I could use different
> format for the Mapper and Combiner. I am using the Text as the format of the
> Mapper and MapWritable as the format of the format.
>
>
>
> But it looks the hadoop didn't support that yet?
>
>
>
> I have some code like the following:
>
>
>
> public class RawLogMapper extends Mapper<LongWritable, Text, Text, Text> {
>
>
>
> public class RawLogCombiner extends Reducer<Text, Text, Text, MapWritable>
> {
>
>
>
> job.setMapOutputKeyClass(Text.class);
>
> job.setMapOutputValueClass(Text.class);
>
>
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(MapWritable.class);
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
>
> But it failed and the logs told me that there are type mismatch. Is there
> anyway I could use different type for the VALUEOUT for the mapper and
> combiner?
>
>
>
> Thanks
>
>
>
>
> Best wishes,
> Xu Wenhao
>

Mime
View raw message