hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James R. Leek" <le...@llnl.gov>
Subject Re: Hadoop with Multiple Inpus and Outputs
Date Thu, 03 Dec 2009 15:43:56 GMT
Thanks, but removing the combiner doesn't seem to have done anything.  
This is what confuses me though, the only strange thing I'm doing is the 
MultipleOutput stuff.  Why is the problem in the mapper then?  The 
Reducer is where I'm using it...


Amogh Vasekar wrote:
> Hi,
> Please try removing the combiner and running.
> I know that if you use multiple outputs from within a mapper, those <k,v> pairs
are not a part of sort and shuffle phase. Your combiner is same as reducer which uses mos,
and might be an issue on map side. If I'm to take a guess, mos writes to a different file
from default map output, and the default key format is LongWritable. If nothing is written,
maybe this isnt modified? Just a thought.
> For checking input file being consumed in current map task, you can use "map.input.file"
from job conf, instead of figuring it out from split name.
> Amogh

View raw message