Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CALwhT97WwVjw8CF_pg-df8JmjLfq7dUqN1Tqb1JaJQqGWhLk=g@mail.gmail.com>
References: 
 <CAKRKJ1MFwDbN7Nepik+j=NJ0y_vO7P0Ob_E10dZoBifj6zYtkQ@mail.gmail.com>
	<CALwhT97WwVjw8CF_pg-df8JmjLfq7dUqN1Tqb1JaJQqGWhLk=g@mail.gmail.com>
Date: Tue, 23 Feb 2016 10:58:59 +0530
Message-ID: 
 <CAKRKJ1NLp0gQb900VFzOaPeGZU5JosdU44Aq5=skrANWy3dTXQ@mail.gmail.com>
Subject: Re: YARN Fair Scheduler
From: Prabhu Joseph <prabhujose.gates@gmail.com>
To: yarn-dev@hadoop.apache.org
Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11490122fa72af052c69390a

--001a11490122fa72af052c69390a
Content-Type: text/plain; charset=UTF-8

Hi Karthik,

   Yes all the queues are always active (atleast one job is running at a
time) and thus the fair share of all queue is very less. How to design the
fair scheduler for this kind of case. Do you have some Best Practices to
design the fair-scheduler.xml.

Weights - is the correct way to make critical queues get a bigger share.
How Nesting of queues helps. And few more doubts

1. How to configure minResources of a queue, is the sum of minResources of
all queue should be equal to Total YARN Cluster Resource.
2. What we need to consider when configuring YARN queue for Spark Jobs

Thanks,
Prabhu Joseph


On Tue, Feb 23, 2016 at 10:35 AM, Karthik Kambatla <kasha@cloudera.com>
wrote:

> Hey Prabhu
>
> Are all the 250 queues always active? If not, the actual (instantaneous)
> fairshare used by the scheduler only considers the active queues (i.e.,
> those that have running applications). Otherwise, you can tune your queues
> (weights, nesting etc.) so the critical queues get a bigger share.
>
> Hope that helps.
>
> On Mon, Feb 22, 2016 at 5:07 PM, Prabhu Joseph <prabhujose.gates@gmail.com
> >
> wrote:
>
> > Hi All,
> >
> >    When YARN Fair Scheduler is configured with a parent root and 250
> child
> > queues for a big Cluster having total resource of 10TB and 3000 Cores.
> The
> > fair share of a child queue is very less. Fair Share is Total Cluster
> > resource / total number of child queues. How to design a Fair Scheduler
> > with many like 250 number of queues in such a way, each queue gets more
> > fair share.
> >
> > Is having Nested Queues or configuring weight or any other way to design.
> >
> > Thanks,
> > Prabhu Joseph
> >
>

--001a11490122fa72af052c69390a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Karthik,<br></div><div><br>=C2=A0=C2=A0 Yes all th=
e queues are always active (atleast one job is running at a time) and thus =
the fair share of all queue is very less. How to design the fair scheduler =
for this kind of case. Do you have some Best Practices to design the fair-s=
cheduler.xml.<br><br></div><div>Weights - is the correct way to make critic=
al queues get a bigger share. How Nesting of queues helps. And few more dou=
bts<br><br></div><div>1. How to configure minResources of a queue, is the s=
um of minResources of all queue should be equal to Total YARN Cluster Resou=
rce.<br></div><div>2. What we need to consider when configuring YARN queue =
for Spark Jobs<br></div><div><br></div><div>Thanks,<br></div><div>Prabhu Jo=
seph<br></div><div><br><br><br></div></div><div class=3D"gmail_extra"><br><=
div class=3D"gmail_quote">On Tue, Feb 23, 2016 at 10:35 AM, Karthik Kambatl=
a <span dir=3D"ltr">&lt;<a href=3D"mailto:kasha@cloudera.com" target=3D"_bl=
ank">kasha@cloudera.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex">Hey Prabhu<br>
<br>
Are all the 250 queues always active? If not, the actual (instantaneous)<br=
>
fairshare used by the scheduler only considers the active queues (i.e.,<br>
those that have running applications). Otherwise, you can tune your queues<=
br>
(weights, nesting etc.) so the critical queues get a bigger share.<br>
<br>
Hope that helps.<br>
<br>
On Mon, Feb 22, 2016 at 5:07 PM, Prabhu Joseph &lt;<a href=3D"mailto:prabhu=
jose.gates@gmail.com">prabhujose.gates@gmail.com</a>&gt;<br>
wrote:<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
&gt; Hi All,<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 When YARN Fair Scheduler is configured with a parent root=
 and 250 child<br>
&gt; queues for a big Cluster having total resource of 10TB and 3000 Cores.=
 The<br>
&gt; fair share of a child queue is very less. Fair Share is Total Cluster<=
br>
&gt; resource / total number of child queues. How to design a Fair Schedule=
r<br>
&gt; with many like 250 number of queues in such a way, each queue gets mor=
e<br>
&gt; fair share.<br>
&gt;<br>
&gt; Is having Nested Queues or configuring weight or any other way to desi=
gn.<br>
&gt;<br>
&gt; Thanks,<br>
&gt; Prabhu Joseph<br>
&gt;<br>
</div></div></blockquote></div><br></div>

--001a11490122fa72af052c69390a--