hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Hung <YTHu...@winbond.com>
Subject RE: change yarn application priority
Date Wed, 04 Jun 2014 00:30:59 GMT
I manage to implement the fair scheduler and very happy with the result.
The  way I use it is to create a queue that constraint the number of resources (maxResources)
to be around 70% of total resources, fortunately the fair-scheduler.xml can be modified on-fly
without restarting the yarn cluster.
Thank to every one that provide me with all the information.

From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Tuesday, June 03, 2014 8:13 PM
To: user@hadoop.apache.org
Subject: Re: change yarn application priority

WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption.
 Note that I guess you could raise the one job's priority, and then the other job's priority
so that when a task finishes the other job gets the next slot. However, you're still stuck
waiting and that can take time.)

Here's the rub though... pre-empting a task could have a negative impact on the long running
job.
If you have large jobs and small jobs, you could create multiple queues and leave some capacity
to the smaller job queue.
So you need to specify the queue up front.  (Its been a while, but can you move jobs across
queues? )

I would suggest that you look at fair scheduler, but also consider multiple queues under capacity
scheduler. You may have admin jobs you want to run in the background while other tasks are
running.


On Jun 2, 2014, at 8:10 PM, Henry Hung <YTHung1@winbond.com<mailto:YTHung1@winbond.com>>
wrote:


@Rohith Sharma

Thank you for the confirmation.
I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways to
do it: FairScheduler or CapacityScheduler.
Apparently from google result, there are more article mentions that FairScheduler is better
than CapacityScheduler.
So, I intend to use FairScheduler first, if you have any more suggestion, please let me know,
thank you again.

Best regards,
Henry

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Friday, May 30, 2014 5:49 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: change yarn application priority

Hi

   Currently there is no provision for changing application priority within the same queue.
 Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.

One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
Steps to be follow is
1.       Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
2.       Enable scheduler monitor
yarn.resourcemanager.scheduler.monitor.enable = true

One job you submit to queue 1 which run 2hours. Another job you submit queue 2.

Hope this will help you.

Thanks & Regards
Rohith Sharma K S


This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by
phone or email immediately and delete it!

From: Henry Hung [mailto:YTHung1@winbond.com]
Sent: 30 May 2014 11:53
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: change yarn application priority

HI All,

I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer) and
will need 4 hours to finish.
Let's say I need to run another application that will be quicker to finish (30 minutes) and
only need 1 Map and 1 Reducer.
If I just execute the new application, it will be in queue waiting for the 1st application
to finish.
Is there a way to change the 2nd application priority to higher than the 1st and let resourcemanager
immediately execute the 2ndapplication?

I'm using Hadoop-2.2.0.

Best regards,
Henry

________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.


________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

Mime
View raw message