hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Setting number of parallel Reducers and Mappers for optimal performance
Date Sat, 11 Aug 2012 02:34:21 GMT
Pavan,

A very important factor is how much CPU and how many spindles you have...

Your proposal for memory (44G in all) seems reasonable.

However, if you have 12 spindles and sufficient CPU I'd do something like 10 or 12 maps of
2G each and 6 reduces with 3G/4G each depending on how you want to slice/dice your slots.

Arun

On Aug 10, 2012, at 1:24 PM, Pavan Kulkarni wrote:

> Hi,
> 
> I was trying to optimize Hadoop-1.0.2 performance by setting
> *mapred.tasktracker.map.tasks.maximum
> ,**mapred.tasktracker.reduce.tasks.maximum*
> such that the entire memory is utilized. The tuning of this parameter is
> given as (CPUS > 2) ? (CPUS * 0.50): 1 for reduce and (CPUS > 2) ? (CPUS *
> 0.75): 1 for map.
> I didn't quite get how they made this suggestion ?  Isn't the setting
> dependent on  main memory available?
> For example I had 48GB of memory and I split the parameters as 32 for
> mappers and 12 for reducers and remaining 4 for OS and other processes.
> Please correct me if my assumption is wrong.Also suggest a way to get the
> optimal performance by setting these parameters. Thanks.
> 
> -- 
> 
> --With Regards
> Pavan Kulkarni

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message