Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of acm@hortonworks.com
 designates 209.85.210.48 as permitted sender)
From: Arun C Murthy <acm@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-12--351487778
Subject: Re: Memory based scheduling
Date: Tue, 30 Oct 2012 10:24:26 -0700
In-Reply-To: 
 <CANY30CNkOw8B2D2cKX9RmTXELyKw1cZrq+ZvuOJDwen_q8HiWA@mail.gmail.com>
To: user@hadoop.apache.org
References: 
 <CANY30CNkOw8B2D2cKX9RmTXELyKw1cZrq+ZvuOJDwen_q8HiWA@mail.gmail.com>
Message-Id: <E73DE67B-B579-4248-998F-145248280CA7@hortonworks.com>


--Apple-Mail-12--351487778
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Yes, use the CapacityScheduler and ask for multiple slots per map or =
reduce task:
=
http://hadoop.apache.org/docs/stable/capacity_scheduler.html#Resource+base=
d+scheduling

Arun

On Oct 30, 2012, at 8:49 AM, Marco Z=FChlke wrote:

> Hi,
>=20
> on our cluster our jobs usually satisfied with less than 2 GB of heap =
space.
> so we have on our 8 GB computers 3 maps maximum and on our 16 GB
> computers 4 maps maximum (we only have quad core CPUs and to have
> memory left for reducers). This works very well.
>=20
> But now we have a new kind of jobs. Each mapper requires at lest 4 GB
> of heap space.
>=20
> Is it possible to limit the number of tasks (mapper) per computer to 1 =
or 2 for
> these kinds of jobs ?
>=20
> Regards,
> Marco
>=20

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


--Apple-Mail-12--351487778
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Yes, =
use the CapacityScheduler and ask for multiple slots per map or reduce =
task:<div><a =
href=3D"http://hadoop.apache.org/docs/stable/capacity_scheduler.html#Resou=
rce+based+scheduling">http://hadoop.apache.org/docs/stable/capacity_schedu=
ler.html#Resource+based+scheduling</a></div><div><br></div><div>Arun</div>=
<div><br><div><div><div>On Oct 30, 2012, at 8:49 AM, Marco Z=FChlke =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite">Hi,<br><br>on our cluster our jobs usually satisfied with =
less than 2 GB of heap space.<br>so we have on our 8 GB computers 3 maps =
maximum and on our 16 GB<br>computers 4 maps maximum (we only have quad =
core CPUs and to have<br>
memory left for reducers). This works very well.<br><br>But now we have =
a new kind of jobs. Each mapper requires at lest 4 GB<br>of heap =
space.<br><br>Is it possible to limit the number of tasks (mapper) per =
computer to 1 or 2 for<br>
these kinds of jobs ?<br><br>Regards,<br>Marco<br><br>
</blockquote></div><br><div>
<div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; "><span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; color: rgb(0, 0, 0); font-family: =
Helvetica; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; orphans: 2; =
text-align: -webkit-auto; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">--</div><div style=3D"word-wrap: =
break-word; -webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space; ">Arun C. Murthy</div><div style=3D"word-wrap: =
break-word; -webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space; ">Hortonworks Inc.<br><a =
href=3D"http://hortonworks.com/">http://hortonworks.com/</a><br><br></div>=
</span></div>
</div>
<br></div></div></body></html>=

--Apple-Mail-12--351487778--