hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: setting a different input/output class for combiner function than map and reduce functions
Date Wed, 24 Sep 2008 00:48:34 GMT
> ==map phase==
> input: key = LongWritable value = Text,
> output: key = Text, value = Longwritable
>
> ==combiner==
> input: key = Text, value = iterator<LongWritable>
> output: key = Text, value = Text
>

The combiner is a pure optimization and *cannot* change the output  
types of the map i.e. the combiner output _must_ be <Text,  
LongWritable>.

> ==reduce phase==
> input: key = Text, value = iterator<Text>
> output: key = Text, value = LongWritable
>

You have a mismatch here too - the map-output types *must* be <Text,  
Text> if your application needs the above. Your application is  
assuming that the Combiner will change the types - this is an invalid  
assumption.

The way around this is to get your map-output types to be <Text, Text>.

Arun


Mime
View raw message