Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of pavan.baburao@gmail.com
 designates 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <C17CACDA-E8A8-40EA-A7A9-642FF1A44847@hortonworks.com>
References: 
 <CAPxqZueMBeDTb9uS+0PdjP4DroRg=dLhn2=Zn_X4C17aJ0LuXQ@mail.gmail.com>
	<C17CACDA-E8A8-40EA-A7A9-642FF1A44847@hortonworks.com>
Date: Fri, 10 Aug 2012 21:17:56 -0700
Message-ID: 
 <CAPxqZufmj3+rjMi6dCy1RAOWbgA4N+3POMKiOccMd62jo5-eqg@mail.gmail.com>
Subject: Re: Setting number of parallel Reducers and Mappers for optimal
 performance
From: Pavan Kulkarni <pavan.baburao@gmail.com>
To: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e89a8f502dd0c6955f04c6f5bebb

--e89a8f502dd0c6955f04c6f5bebb
Content-Type: text/plain; charset=ISO-8859-1

Arun,

  Thanks a lot for your response.

I am running on a 16 core Xeon processor and 12 spindles.So running 12
Mappers with 2G and 6 Reducers with 3G might give me the best
performance.Also is there a general formula to arrive at those numbers?

On Fri, Aug 10, 2012 at 7:34 PM, Arun C Murthy <acm@hortonworks.com> wrote:

> Pavan,
>
> A very important factor is how much CPU and how many spindles you have...
>
> Your proposal for memory (44G in all) seems reasonable.
>
> However, if you have 12 spindles and sufficient CPU I'd do something like
> 10 or 12 maps of 2G each and 6 reduces with 3G/4G each depending on how you
> want to slice/dice your slots.
>
> Arun
>
> On Aug 10, 2012, at 1:24 PM, Pavan Kulkarni wrote:
>
> > Hi,
> >
> > I was trying to optimize Hadoop-1.0.2 performance by setting
> > *mapred.tasktracker.map.tasks.maximum
> > ,**mapred.tasktracker.reduce.tasks.maximum*
> > such that the entire memory is utilized. The tuning of this parameter is
> > given as (CPUS > 2) ? (CPUS * 0.50): 1 for reduce and (CPUS > 2) ? (CPUS
> *
> > 0.75): 1 for map.
> > I didn't quite get how they made this suggestion ?  Isn't the setting
> > dependent on  main memory available?
> > For example I had 48GB of memory and I split the parameters as 32 for
> > mappers and 12 for reducers and remaining 4 for OS and other processes.
> > Please correct me if my assumption is wrong.Also suggest a way to get the
> > optimal performance by setting these parameters. Thanks.
> >
> > --
> >
> > --With Regards
> > Pavan Kulkarni
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 

--With Regards
Pavan Kulkarni

--e89a8f502dd0c6955f04c6f5bebb--