hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Armstrong <john.armstr...@ccri.com>
Subject Re: Extending ArrayWritable, Using Combiner and Spill Failed error
Date Thu, 18 Aug 2011 20:51:50 GMT
On Thu, 18 Aug 2011 13:44:22 -0700, vipul sharma
<sharmavipulwork@gmail.com> wrote:
> *I think the error is due to using combiner. Since combiner is output
> in Text and Reducer is expecting IntArrayWritable. If I remove combiner
> everything works. What am I doing wrong and how can I get the combiner
> work? Any help is greatly appreciated.

It's not immediately apparent, but combiners may run "tiered".  That is,
the mapper output may be run through a bunch of combiners, and THEIR output
may be run through another tier of combiners, and so on.  Thus the combiner
input/output types must be exactly the same.  You'll have to write another
Combiner class, which handles the combining, and then have your Reducer
class do the combining AND the conversion into output values.

In the absence of chainreducers, I'd suggest creating an abstract
MyCombinerReducer class which implements the combining algorithm, then
derive both the Combiner and Reducer from it so they can both call the same
combining code.

View raw message