Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of java8964@hotmail.com designates
 65.55.90.82 as permitted sender)
Message-ID: <SNT149-W456A2DE8CE6AF738A347C2D0AC0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_4bbae670-9632-4b91-9c26-3f64249a0a81_"
From: java8964 <java8964@hotmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: Force one mapper per machine (not core)?
Date: Wed, 29 Jan 2014 08:18:37 -0500
Importance: Normal
In-Reply-To: <9EACF911-AFAA-409D-9411-7F9DB2591F9F@keithwiley.com>
References: <280A85F8-DBB1-483D-843D-0D99C3356689@keithwiley.com>
 <CANZDn9vL72ESk0UZPL+wsCYRGpAGPvUyw7g8Sz8HwLD3t+P4fA@mail.gmail.com>,<9EACF911-AFAA-409D-9411-7F9DB2591F9F@keithwiley.com>
MIME-Version: 1.0

--_4bbae670-9632-4b91-9c26-3f64249a0a81_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Or you can implement your own InputSplit and InputFormat=2C which you can c=
ontrol how to send tasks to which node=2C and how many per node.
Some detail examples you can get from book "Professional Hadoop Solution" C=
haracter 4.
Yong

> Subject: Re: Force one mapper per machine (not core)?
> From: kwiley@keithwiley.com
> Date: Tue=2C 28 Jan 2014 15:41:22 -0800
> To: user@hadoop.apache.org
>=20
> Yeah=2C it isn't=2C not even remotely=2C but thanks.
>=20
> On Jan 28=2C 2014=2C at 14:06 =2C Bryan Beaudreault wrote:
>=20
> > If this cluster is being used exclusively for this goal=2C you could ju=
st set the mapred.tasktracker.map.tasks.maximum to 1.
> >=20
> >=20
> > On Tue=2C Jan 28=2C 2014 at 5:00 PM=2C Keith Wiley <kwiley@keithwiley.c=
om> wrote:
> > I'm running a program which in the streaming layer automatically multit=
hreads and does so by automatically detecting the number of cores on the ma=
chine.  I realize this model is somewhat in conflict with Hadoop=2C but non=
etheless=2C that's what I'm doing.  Thus=2C for even resource utilization=
=2C it would be nice to not only assign one mapper per core=2C but only one=
 mapper per machine.  I realize that if I saturate the cluster none of this=
 really matters=2C but consider the following example for clarity: 4-core n=
odes=2C 10-node cluster=2C thus 40 slots=2C fully configured across mappers=
 and reducers (40 slots of each).  Say I run this program with just two map=
pers.  It would run much more efficiently (in essentially half the time) if=
 I could force the two mappers to go to slots on two separate machines inst=
ead of running the risk that Hadoop may assign them both to the same machin=
e.
> >=20
> > Can this be done?
> >=20
> > Thanks.
>=20
> _________________________________________________________________________=
_______
> Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwi=
ley.com
>=20
> "Luminous beings are we=2C not this crude matter."
>                                            --  Yoda
> _________________________________________________________________________=
_______
>=20
 		 	   		  =

--_4bbae670-9632-4b91-9c26-3f64249a0a81_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style><!--
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 12pt=3B
font-family:Calibri
}
--></style></head>
<body class=3D'hmmessage'><div dir=3D'ltr'>Or you can implement your own In=
putSplit and InputFormat=2C which you can control how to send tasks to whic=
h node=2C and how many per node.<div><br></div><div>Some detail examples yo=
u can get from book "Professional Hadoop Solution" Character 4.</div><div><=
br></div><div>Yong<br><br><div>&gt=3B Subject: Re: Force one mapper per mac=
hine (not core)?<br>&gt=3B From: kwiley@keithwiley.com<br>&gt=3B Date: Tue=
=2C 28 Jan 2014 15:41:22 -0800<br>&gt=3B To: user@hadoop.apache.org<br>&gt=
=3B <br>&gt=3B Yeah=2C it isn't=2C not even remotely=2C but thanks.<br>&gt=
=3B <br>&gt=3B On Jan 28=2C 2014=2C at 14:06 =2C Bryan Beaudreault wrote:<b=
r>&gt=3B <br>&gt=3B &gt=3B If this cluster is being used exclusively for th=
is goal=2C you could just set the mapred.tasktracker.map.tasks.maximum to 1=
.<br>&gt=3B &gt=3B <br>&gt=3B &gt=3B <br>&gt=3B &gt=3B On Tue=2C Jan 28=2C =
2014 at 5:00 PM=2C Keith Wiley &lt=3Bkwiley@keithwiley.com&gt=3B wrote:<br>=
&gt=3B &gt=3B I'm running a program which in the streaming layer automatica=
lly multithreads and does so by automatically detecting the number of cores=
 on the machine.  I realize this model is somewhat in conflict with Hadoop=
=2C but nonetheless=2C that's what I'm doing.  Thus=2C for even resource ut=
ilization=2C it would be nice to not only assign one mapper per core=2C but=
 only one mapper per machine.  I realize that if I saturate the cluster non=
e of this really matters=2C but consider the following example for clarity:=
 4-core nodes=2C 10-node cluster=2C thus 40 slots=2C fully configured acros=
s mappers and reducers (40 slots of each).  Say I run this program with jus=
t two mappers.  It would run much more efficiently (in essentially half the=
 time) if I could force the two mappers to go to slots on two separate mach=
ines instead of running the risk that Hadoop may assign them both to the sa=
me machine.<br>&gt=3B &gt=3B <br>&gt=3B &gt=3B Can this be done?<br>&gt=3B =
&gt=3B <br>&gt=3B &gt=3B Thanks.<br>&gt=3B <br>&gt=3B _____________________=
___________________________________________________________<br>&gt=3B Keith=
 Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com=
<br>&gt=3B <br>&gt=3B "Luminous beings are we=2C not this crude matter."<br=
>&gt=3B                                            --  Yoda<br>&gt=3B _____=
___________________________________________________________________________=
<br>&gt=3B <br></div></div> 		 	   		  </div></body>
</html>=

--_4bbae670-9632-4b91-9c26-3f64249a0a81_--