hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Buell <jbu...@vmware.com>
Subject Re: Will hadoop always spread the work evenly between nodes?
Date Wed, 13 Mar 2013 20:47:40 GMT
I think in your case it will have to be even, because all the slots will get filled. A more
interesting case is if you have 40 nodes, will you get exactly 5 slots used for each of the
nodes? Or will some nodes get more than 5 mappers, and others less? I don't remember the details,
but I've had problems with unevenness in such scenarios. At least in MR1, you can usually
force evenness by adjusting the number of map and reduce slots per node. In MR2 the slots
are combined so achieving evenness will be more difficult. 


----- Original Message -----

From: "jeremy p" <athomewithagroovebox@gmail.com> 
To: user@hadoop.apache.org 
Sent: Wednesday, March 13, 2013 1:01:46 PM 
Subject: Will hadoop always spread the work evenly between nodes? 

Say I have 200 input files and 20 nodes, and each node has 10 mapper slots. Will Hadoop always
allocate the work evenly, such that each node will get 10 input files and simultaneously start
10 mappers? Is there a way to force this behavior? 


View raw message