hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Jennings III <raymondj...@yahoo.com>
Subject Re: Hadoop does not follow my setting
Date Thu, 22 Apr 2010 16:55:08 GMT
Isn't the number of mappers specified "only a suggestion" ?

--- On Thu, 4/22/10, He Chen <airbots@gmail.com> wrote:

> From: He Chen <airbots@gmail.com>
> Subject: Hadoop does not follow my setting
> To: common-user@hadoop.apache.org
> Date: Thursday, April 22, 2010, 12:50 PM
> Hi everyone
> 
> I am doing a benchmark by using Hadoop 0.20.0's wordcount
> example. I have a
> 30GB file. I plan to test differenct number of mappers'
> performance. For
> example, for a wordcount job, I plan to test 22 mappers, 44
> mappers, 66
> mappers and 110 mappers.
> 
> However, I set the "mapred.map.tasks" equals to 22. But
> when I ran the job,
> it shows 436 mappers total.
> 
> I think maybe the wordcount set its parameters inside the
> its own program. I
> give "-Dmapred.map.tasks=22" to this program. But it is
> still 436 again in
> my another try.  I found out that 30GB divide by 436
> is just 64MB, it is
> just my block size.
> 
> Any suggestions will be appreciated.
> 
> Thank you in advance!
> 
> -- 
> Best Wishes!
> 顺送商祺!
> 
> --
> Chen He
> (402)613-9298
> PhD. student of CSE Dept.
> Holland Computing Center
> University of Nebraska-Lincoln
> Lincoln NE 68588
> 


      

Mime
View raw message