hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject Re: How does hadoop decide how many reducers to run?
Date Sat, 12 Jan 2013 14:05:40 GMT
Since you are using EMR,  AWS pre configures the number of slots per node. 
So you are already getting the optimum number of slots that their 'machines' can handle. 

So when you run your job, you said that you saw 16 reducers and then 12 reducers running.


This could imply that your job required 28 reducers  and it was using the full resources of
the machines. 

On Jan 11, 2013, at 5:53 PM, Roy Smith <roy@panix.com> wrote:

> On Jan 11, 2013, at 6:20 PM, Michael Segel wrote:
> 
>> Hi, 
>> 
>> First, not enough information. 
>> 
>> 1) EC2 got it. 
>> 2) Which flavor of Hadoop? Is this EMR as well? 
> 
> Yes, EMR.  We're running AMI version 2.3.1, which includes hadoop 1.0.3.
> 
> 
>> 3) How many slots did you configure in your mapred-site.xml?
> 
> Hmmm, no clue.  I've never even heard of that file.  We're using mrjob.  It may be that
mrjob is building a mapred-site.xml file for me and I never even see it?
> 
> ---
> Roy Smith
> roy@panix.com
> 
> 
> 
> 


Mime
View raw message