hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mostafa Gaber <moustafa.ga...@gmail.com>
Subject Re: Reducer IO
Date Mon, 06 Feb 2012 16:50:47 GMT
Hello Marek,

I think you can increase number of reducers for your MR job so as to reduce
the amount of intermediate key-value pairs assigned to each reducer. Note
also that the number of reducers is dependent on your job and how the
output should be produced.

On Mon, Feb 6, 2012 at 11:37 AM, Marek Miglinski <mmiglinski@seven.com>wrote:

> Hey,
> I have a mapreduce job (transactions loader) and the main problem of it is
> "reduce->copy" and "reduce->sort" phase which takes all IO and uses all
> disk resources, what are the possible ways to reduce this load? My cloud
> settings are:
> ioSortFactor=80
> ioSortMb=800
> (mapredChildJavaOpts=Xmx1152m)
> I can lower those settings, what else can I tweak?
> Thanks,
> Marek M.

Best Regards,
Mostafa Ead

View raw message