hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <msegel_had...@hotmail.com>
Subject Re: change yarn application priority
Date Tue, 03 Jun 2014 12:12:52 GMT
WRT capacity scheduler, its not so much changing the priority of a job, but allowing for pre-emption.
 Note that I guess you could raise the one job's priority, and then the other job's priority
so that when a task finishes the other job gets the next slot. However, you're still stuck
waiting and that can take time.) 

Here's the rub though... pre-empting a task could have a negative impact on the long running
job. 
If you have large jobs and small jobs, you could create multiple queues and leave some capacity
to the smaller job queue. 
So you need to specify the queue up front.  (Its been a while, but can you move jobs across
queues? )

I would suggest that you look at fair scheduler, but also consider multiple queues under capacity
scheduler. You may have admin jobs you want to run in the background while other tasks are
running. 

 
On Jun 2, 2014, at 8:10 PM, Henry Hung <YTHung1@winbond.com> wrote:

> @Rohith Sharma
>  
> Thank you for the confirmation.
> I already google some information regarding scheduler in Hadoop 2.2.0, there are 2 ways
to do it: FairScheduler or CapacityScheduler.
> Apparently from google result, there are more article mentions that FairScheduler is
better than CapacityScheduler.
> So, I intend to use FairScheduler first, if you have any more suggestion, please let
me know, thank you again.
>  
> Best regards,
> Henry
>  
> From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com] 
> Sent: Friday, May 30, 2014 5:49 PM
> To: user@hadoop.apache.org
> Subject: RE: change yarn application priority
>  
> Hi
>  
>    Currently there is no provision for changing application priority within the same
queue.  Follow the Jirahttps://issues.apache.org/jira/i#browse/YARN-1963 for this new feature.
>  
> One way you can achieve by using enabling scheduler monitors for CapacitySchedulers.
> Steps to be follow is
> 1.       Configure 2 queues, follow http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> 2.       Enable scheduler monitor
> yarn.resourcemanager.scheduler.monitor.enable = true
>  
> One job you submit to queue 1 which run 2hours. Another job you submit queue 2.
>  
> Hope this will help you.
>  
> Thanks & Regards
> Rohith Sharma K S
>  
>  
> This e-mail and its attachments contain confidential information from HUAWEI, which 
> is intended only for the person or entity whose address is listed above. Any use of the

> information contained herein in any way (including, but not limited to, total or partial

> disclosure, reproduction, or dissemination) by persons other than the intended 
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by 
> phone or email immediately and delete it!
>  
> From: Henry Hung [mailto:YTHung1@winbond.com] 
> Sent: 30 May 2014 11:53
> To: user@hadoop.apache.org
> Subject: change yarn application priority
>  
> HI All,
>  
> I have an application that consumes all of nodemanager capacity (30 Map and 1 Reducer)
and will need 4 hours to finish.
> Let’s say I need to run another application that will be quicker to finish (30 minutes)
and only need 1 Map and 1 Reducer.
> If I just execute the new application, it will be in queue waiting for the 1st application
to finish.
> Is there a way to change the 2nd application priority to higher than the 1st and let
resourcemanager immediately execute the 2ndapplication?
>  
> I’m using Hadoop-2.2.0.
>  
> Best regards,
> Henry
>  
> The privileged confidential information contained in this email is intended for use only
by the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.
> 
> The privileged confidential information contained in this email is intended for use only
by the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.



Mime
View raw message