hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Hung <YTHu...@winbond.com>
Subject RE: change yarn application priority
Date Tue, 03 Jun 2014 01:10:10 GMT
@Rohith Sharma

Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to
do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better
than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know,
thank you again.

Best regards,
Henry

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org
Subject: RE: change yarn application priority

Hi

   Currently there is no provision for changing application priority within the same queue.
 Follow the Jira https://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.

One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is

1.       Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

2.       Enable scheduler monitor

yarn.resourcemanager.scheduler.monitor.enable = true

One job you submit to queue 1 which run 2hours. Another job you submit queue 2.

Hope this will help you.

Thanks & Regards
Rohith Sharma K S


This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by
phone or email immediately and delete it!

From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: change yarn application priority

HI All,

I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and
will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and
only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application
to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager
immediately execute the 2nd application?

I'm using Hadoop-2.2.0.

Best regards,
Henry

________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

Mime
View raw message