Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B8749115BC for ; Wed, 13 Aug 2014 17:06:48 +0000 (UTC) Received: (qmail 47206 invoked by uid 500); 13 Aug 2014 17:06:43 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 47071 invoked by uid 500); 13 Aug 2014 17:06:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 47060 invoked by uid 99); 13 Aug 2014 17:06:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 17:06:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of y.z.elshater@gmail.com designates 209.85.215.51 as permitted sender) Received: from [209.85.215.51] (HELO mail-la0-f51.google.com) (209.85.215.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 17:06:38 +0000 Received: by mail-la0-f51.google.com with SMTP id pn19so27549lab.24 for ; Wed, 13 Aug 2014 10:06:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=xLX8nLLlCSfVmJCB32jvmU7JLakQLVtCARFI+YcTyyI=; b=uNiqPh5aA/h3Sn6mffBgGNFijby42hwaBKEsVtMdTuxuLSlMa992JhVxTwbLOTPmds 6PWSy2L5A810SdDhlP7eb7vObzGniWQSfQYs8WCJ6I+iuxcbFb8nsEARVgLSBP5sT3cP GLzXOcf1eXXNFOo6ugZPUUkXF7mvUodZGGix3F6Krn1yk9uqr7B+Xm23p/nuLg0pDx3L GaAFoUAU/G0uixXnU+MPaS9tXpOr5F7b+ImdjWHdLQR9dJ07d7x7Jv3/Dyb0vqvkBAZO 7tsUCvQ7PqlIF/F8RK0wTWXsC0/eNvhh31c00CtgzZiPg96txx/lZv2Cj9lwBXgDZxrP lilQ== MIME-Version: 1.0 X-Received: by 10.152.9.100 with SMTP id y4mr5961119laa.26.1407949577393; Wed, 13 Aug 2014 10:06:17 -0700 (PDT) Received: by 10.112.1.7 with HTTP; Wed, 13 Aug 2014 10:06:17 -0700 (PDT) In-Reply-To: <53DC189E5FAEFA43BFA1BC02431031DB740DFF4883@WECTMLBOX.winbond.com.tw> References: <53DC189E5FAEFA43BFA1BC02431031DB740DF31CBA@WECTMLBOX.winbond.com.tw> <53DC189E5FAEFA43BFA1BC02431031DB740DFF4883@WECTMLBOX.winbond.com.tw> Date: Wed, 13 Aug 2014 13:06:17 -0400 Message-ID: Subject: Re: fair scheduler not working as intended From: Yehia Elshater To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0158b9c86c1c75050085cec3 X-Virus-Checked: Checked by ClamAV on apache.org --089e0158b9c86c1c75050085cec3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Henry, I think FairScheduler is a better option for your case. As if you used FifoScheduler, the latency of the short jobs will be worst (in case of any of longrun jobs are currently running in your cluster). So I think your queue layout is good to apply fairness between the long and short jobs. Long run queue will gain more resources (more than its min share) just in case that the short one does not have any jobs to run in the mean time. These additional resources will be diminished gradually to its minimum share again based on the short queue workload to ensure fairness. I hope that helps you. Yehia On 13 August 2014 02:01, Henry Hung wrote: > Hi Yehia, > > > > Oh? I thought that by using maxResources =3D 15360 mb (3072 mb * 5), vcor= es > =3D 5, and maxMaps =3D 5, I already restricting the job to only use 5 map= s at > max. > > > > The reason is my long run job have 841 maps, and each map will process > data for almost 2 hours. > > In the meantime there will be some short jobs that only need couple of > minutes to complete. > > Hence why I use fair scheduler to split resources into 2 groups, one > default and other one longrun. > > I want to make sure there always an available resources ready to be used > by short jobs. > > > > If your explanation is true, then current fair scheduler behavior is not > what I wanted. > > So is there any other ways to setup YARN resources to accommodate the > short / long run jobs? > Or do I need to create 2 separate YARN cluster? (I have been thinking > about this approach) > > > > Best regards, > > Henry > > > > *From:* Yehia Elshater [mailto:y.z.elshater@gmail.com] > *Sent:* Wednesday, August 13, 2014 11:27 AM > *To:* user@hadoop.apache.org > *Subject:* Re: fair scheduler not working as intended > > > > Hi Henry, > > > > Are there any applications (on different queues rather than longrun queue= ) > are running in the same time ? I think FairScheduler is going to assign > more resources to your "longrun" as long as there no other applications a= re > running in the other queues. > > > > Thanks > > Yehia > > > > On 12 August 2014 20:30, Henry Hung wrote: > > Hi Everyone, > > > > I=E2=80=99m using Hadoop-2.2.0 with fair scheduler in my YARN cluster, bu= t > something is wrong with the fair scheduler. > > > > Here is my fair-scheduler.xml looks like: > > > > > > > > 15360 mb, 5 vcores > > 0.5 > > 2 > > 5 > > 1 > > > > > > > > I create a =E2=80=9Clongrun=E2=80=9D queue to ensure that huge MR applica= tion can only use > 5 resources. My YARN setup for each resource memory is 3072 MB: > > > > > > mapreduce.map.memory.mb > > 3072 > > > > > > mapreduce.reduce.memory.mb > > 3072 > > > > > > When the huge application started, it works just fine and scheduler > restrict it to only run 5 maps in parallel. > > But after running for some time, the application run 10 maps in parallel. > > The scheduler page show that the =E2=80=9Clongrun=E2=80=9D queue used 66%= , exceed the fair > share 30%. > > > > Can anyone tell me why the application can get more than it deserved? > > Is the problem with my configuration? Or there is a bug? > > > > Best regards, > > Henry Hung > > > ------------------------------ > > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original send= er > of this email. If you are not the addressee indicated in this email or ar= e > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in thi= s > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. > > > > ------------------------------ > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original send= er > of this email. If you are not the addressee indicated in this email or ar= e > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in thi= s > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. > --089e0158b9c86c1c75050085cec3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Henry,

I think FairScheduler is a be= tter option for your case. As if you used FifoScheduler, the latency of the= short jobs will be worst (in case of any of longrun jobs are currently run= ning in your cluster). So I think your queue layout is good to apply fairne= ss between the long and short jobs. Long run queue will gain more resources= (more than its min share) just in case that the short one does not have an= y jobs to run in the mean time. These additional resources will be diminish= ed gradually to its minimum share again based on the short queue workload t= o ensure fairness.

I hope that helps you.

Yehia


On = 13 August 2014 02:01, Henry Hung <YTHung1@winbond.com> wro= te:

Hi Yehia,

=C2=A0

Oh? I thought that by usi= ng maxResources =3D 15360 mb (3072 mb * 5), vcores =3D 5, and maxMaps =3D 5= , I already restricting the job to only use 5 maps at max.

=C2=A0

The reason is my long run= job have 841 maps, and each map will process data for almost 2 hours.

In the meantime there wil= l be some short jobs that only need couple of minutes to complete.

Hence why I use fair sche= duler to split resources into 2 groups, one default and other one longrun.<= u>

I want to make sure there= always an available resources ready to be used by short jobs.

=C2=A0

If your explanation is tr= ue, then current fair scheduler behavior is not what I wanted.

So is there any other way= s to setup YARN resources to accommodate the short / long run jobs?
Or do I need to create 2 separate YARN cluster? (I have been thinking about= this approach)

=C2=A0

Best regards,

Henry

=C2=A0

From: Yehia = Elshater [mailto:y.z.elshater@gmail.com]
Sent: Wednesday, August 13, 2014 11:27 AM
To: user= @hadoop.apache.org
Subject: Re: fair scheduler not working as intended

=C2=A0

Hi Henry,=

=C2=A0

Are there any applicati= ons (on different queues rather than longrun queue) are running in the same= time ? I think FairScheduler is going to assign more resources to your "longrun" as long as there no other applications are run= ning in the other queues.

=C2=A0

Thanks=

Yehia<= /u>

=C2=A0<= /p>

On 12 August 2014 20:30, Henry Hung <YTHung1@winbond.com> wr= ote:

Hi Everyone,

=C2=A0

I=E2=80=99m using Hadoop-2.2.0 with fair scheduler i= n my YARN cluster, but something is wrong with the fair scheduler.

=C2=A0

Here is my fair-scheduler.xml looks like:<= /u>

=C2=A0

<allocations>

=C2=A0 <queue name=3D"longrun">

=C2=A0=C2=A0=C2=A0 <maxResources>15360 mb, 5 v= cores</maxResources>

=C2=A0=C2=A0=C2=A0 <weight>0.5</weight><= u>

=C2=A0=C2=A0=C2=A0 <minMaps>2</minMaps><= u>

=C2=A0=C2=A0=C2=A0 <maxMaps>5</maxMaps><= u>

=C2=A0=C2=A0=C2=A0 <minReduces>1</minReduce= s>

=C2=A0 </queue>

</allocations>

=C2=A0

I create a =E2=80=9Clongrun=E2=80=9D queue to ensure= that huge MR application can only use 5 resources. My YARN setup for each = resource memory is 3072 MB:

=C2=A0

=C2=A0 <property>

=C2=A0=C2=A0=C2=A0 <name>mapreduce.map.memory.= mb</name>

=C2=A0=C2=A0=C2=A0 <value>3072</value>

=C2=A0 </property>

=C2=A0 <property>

=C2=A0=C2=A0=C2=A0 <name>mapreduce.reduce.memo= ry.mb</name>

=C2=A0=C2=A0=C2=A0 <value>3072</value>

=C2=A0 </property>

=C2=A0

When the huge application started, it works just fin= e and scheduler restrict it to only run 5 maps in parallel.

But after running for some time, the application run= 10 maps in parallel.

The scheduler page show that the =E2=80=9Clongrun=E2= =80=9D queue used 66%, exceed the fair share 30%.

=C2=A0

Can anyone tell me why the application can get more = than it deserved?

Is the problem with my configuration? Or there is a = bug?

=C2=A0

Best regards,

Henry Hung

=C2=A0

The privileged confidential inf= ormation contained in this email is intended for use only by the addressees= as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for d= elivery of the email to such a person, please kindly reply to the sender in= dicating this fact and delete all copies of it from your computer and netwo= rk server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confiden= tial information of Winbond is strictly prohibited; and any information in = this email irrelevant to the official business of Winbond shall be deemed a= s neither given nor endorsed by Winbond.

=C2=A0


The privileged confidential = information contained in this email is intended for use only by the address= ees as indicated by the original sender of this email. If you are not the a= ddressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly = reply to the sender indicating this fact and delete all copies of it from y= our computer and network server immediately. Your cooperation is highly app= reciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohib= ited; and any information in this email irrelevant to the official business= of Winbond shall be deemed as neither given nor endorsed by Winbond.

--089e0158b9c86c1c75050085cec3--