Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Wed, 13 Mar 2013 13:47:40 -0700 (PDT)
From: Jeffrey Buell <jbuell@vmware.com>
To: user@hadoop.apache.org
Message-ID: <42273428.14551265.1363207660022.JavaMail.root@vmware.com>
In-Reply-To: 
 <CAKjW-3JaR2xqNdXsrZsMB4LvBYJYLg8UfYE557_+B9kTZwz1HQ@mail.gmail.com>
Subject: Re: Will hadoop always spread the work evenly between nodes?
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_14551264_2102637780.1363207660021"

------=_Part_14551264_2102637780.1363207660021
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I think in your case it will have to be even, because all the slots will get filled. A more interesting case is if you have 40 nodes, will you get exactly 5 slots used for each of the nodes? Or will some nodes get more than 5 mappers, and others less? I don't remember the details, but I've had problems with unevenness in such scenarios. At least in MR1, you can usually force evenness by adjusting the number of map and reduce slots per node. In MR2 the slots are combined so achieving evenness will be more difficult. 

Jeff 

----- Original Message -----

From: "jeremy p" <athomewithagroovebox@gmail.com> 
To: user@hadoop.apache.org 
Sent: Wednesday, March 13, 2013 1:01:46 PM 
Subject: Will hadoop always spread the work evenly between nodes? 

Say I have 200 input files and 20 nodes, and each node has 10 mapper slots. Will Hadoop always allocate the work evenly, such that each node will get 10 input files and simultaneously start 10 mappers? Is there a way to force this behavior? 


--Jeremy 

------=_Part_14551264_2102637780.1363207660021
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><style type=3D'text/css'>p { margin: 0; }</style></head><body><=
div style=3D'font-family: tahoma,new york,times,serif; font-size: 12pt; col=
or: #000000'>I think in your case it will have to be even, because all the =
slots will get filled.&nbsp; A more interesting case is if you have 40 node=
s, will you get exactly 5 slots used for each of the nodes?&nbsp; Or will s=
ome nodes get more than 5 mappers, and others less?&nbsp; I don't remember =
the details, but I've had problems with unevenness in such scenarios.&nbsp;=
 At least in MR1, you can usually force evenness by adjusting the number of=
 map and reduce slots per node.&nbsp; In MR2 the slots are combined so achi=
eving evenness will be more difficult.<br><br>Jeff<br><br><hr id=3D"zwchr">=
<div style=3D"color:#000;font-weight:normal;font-style:normal;text-decorati=
on:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </=
b>"jeremy p" &lt;athomewithagroovebox@gmail.com&gt;<br><b>To: </b>user@hado=
op.apache.org<br><b>Sent: </b>Wednesday, March 13, 2013 1:01:46 PM<br><b>Su=
bject: </b>Will hadoop always spread the work evenly between nodes?<br><br>=
Say I have 200 input files and 20 nodes, and each node has 10 mapper slots.=
 &nbsp;Will Hadoop always allocate the work evenly, such that each node wil=
l get 10 input files and simultaneously start 10 mappers? &nbsp;Is there a =
way to force this behavior?<div>

<br></div><div>--Jeremy</div>
</div><br></div></body></html>
------=_Part_14551264_2102637780.1363207660021--