Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sandy.ryza@cloudera.com
 designates 209.85.210.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <3D527B17-E3EC-4988-BEEC-D3CAF8448658@oit.gatech.edu>
References: <3D527B17-E3EC-4988-BEEC-D3CAF8448658@oit.gatech.edu>
Date: Tue, 21 May 2013 14:57:41 -0700
Message-ID: 
 <CACBYxK+yg6W6ijPtk3hwaqCe-fKMOwrPV3TF5qxwYGbOOeG-uQ@mail.gmail.com>
Subject: Re: Is there a way to limit # of hadoop tasks per user at runtime?
From: Sandy Ryza <sandy.ryza@cloudera.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec544f02ec9f31604dd4189e1

--bcaec544f02ec9f31604dd4189e1
Content-Type: text/plain; charset=ISO-8859-1

Hi Mehmet,

Are you using MR1 or MR2?

The fair scheduler, present in both versions, but configured slightly
differently, allows you to limit the number of map and reduce tasks in a
queue.  The configuration can be updated at runtime by modifying the
scheduler's allocations file.  It also has a feature that automatically
maps jobs to queues based on the user submitted them.

Here are links to documentation in MR1 and MR2:
http://hadoop.apache.org/docs/stable/fair_scheduler.html
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

-Sandy


On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu
> wrote:

> Hi Everyone,
>
> I was wondering if there is a way for limiting the number of tasks
> (map+reduce) *per user* at runtime? Using an environment variable perhaps?
> I am asking this from a resource provisioning perspective. I am trying to
> come up with a N-token licensing system for multiple users to use our
> limited hadoop resources simultaneously. That is, when user A checks out 6
> tokens,  he/she can only run 6 hadoop tasks.
>
> If there is no such thing in hadoop, has anyone tried to integrate hadoop
> with torque/moab (or any other RM or scheduler)? Any advice in that
> direction will be appreciated :)
>
> Thanks in advance,
> -Mehmet
>
>
>
>
>
>
>

--bcaec544f02ec9f31604dd4189e1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Mehmet,<div><br></div><div style>Are you using MR1 or M=
R2?</div><div style><br></div><div style>The fair scheduler, present in bot=
h versions, but configured slightly differently, allows you to limit the nu=
mber of map and reduce tasks in a queue. =A0The configuration can be update=
d at runtime by modifying the scheduler&#39;s allocations file. =A0It also =
has a feature that automatically maps jobs to queues based on the user subm=
itted them.</div>
<div style><br></div><div style>Here are links to documentation in MR1 and =
MR2:</div><div style><a href=3D"http://hadoop.apache.org/docs/stable/fair_s=
cheduler.html">http://hadoop.apache.org/docs/stable/fair_scheduler.html</a>=
<br>
</div><div style><a href=3D"http://hadoop.apache.org/docs/current/hadoop-ya=
rn/hadoop-yarn-site/FairScheduler.html">http://hadoop.apache.org/docs/curre=
nt/hadoop-yarn/hadoop-yarn-site/FairScheduler.html</a><br></div><div style>
<br></div><div style>-Sandy</div><div style><br></div></div><div class=3D"g=
mail_extra"><br><br><div class=3D"gmail_quote">On Tue, May 21, 2013 at 2:43=
 PM, Mehmet Belgin <span dir=3D"ltr">&lt;<a href=3D"mailto:mehmet.belgin@oi=
t.gatech.edu" target=3D"_blank">mehmet.belgin@oit.gatech.edu</a>&gt;</span>=
 wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word">Hi Every=
one,<div><br></div><div>I was wondering if there is a way for limiting the =
number of tasks (map+reduce) *per user* at runtime? Using an environment va=
riable perhaps? I am asking this from a resource provisioning perspective. =
I am trying to come up with a N-token licensing system for multiple users t=
o use our limited hadoop resources simultaneously. That is, when user A che=
cks out 6 tokens, =A0he/she can only run 6 hadoop tasks.=A0</div>
<div><br></div><div>If there is no such thing in hadoop, has anyone tried t=
o integrate hadoop with torque/moab (or any other RM or scheduler)? Any adv=
ice in that direction will be appreciated :)</div><div><br></div><div>Thank=
s in advance,</div>
<div>-Mehmet</div><div><br><div><br></div><div><br></div><div><div><span st=
yle=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-align=
:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;borde=
r-collapse:separate;text-transform:none;font-size:medium;white-space:normal=
;font-family:Helvetica;word-spacing:0px"><br>
</span><br>
</div>
<br></div></div></div></blockquote></div><br></div>

--bcaec544f02ec9f31604dd4189e1--