hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wuchang <583424...@qq.com>
Subject Re: How to make a queue cannot be preempted under FairScheduler?
Date Mon, 15 May 2017 08:23:33 GMT
Great thanks. 
It seems that for my Hadoop 2.7.3 it’s absolutely  impossible to prevent a queue from being
preempted.But I have tried to make a queue which is more important a higher fairSharePreemptionThreshold
value , and a queue which is less important a lower  fairSharePreemptionThreshold  value 
, I believe this will make the queue which is more important to be easier to preempt the resources
of queue which is less important, thus the resource of important queue is harder to be preempted
by other unimportant queues;

I don’ t know if this is just a workaround.

> On May 15, 2017, at 16:08, Sasaki Kai <sasaki@treasure-data.com> wrote:
> 
> Hi
> 
> I believe there is no nice way to prevent preemption from specific queue. 
> Only thing we can do is disabling preemption completely.
> 
> <property>
>     <name>yarn.scheduler.fair.preemption</name>
>     <value>true</value>
> </property>
> 
> From Hadoop 2.9.0 or 3.0.0-alpha1 we will be able to use allowPreemptionFrom property
to forbid preemption from the queue.
> 
> <queue name=“highPriority">
>     <allowPreemptionFrom>false</allowPreemptionFrom>
> </queue>
> 
> https://issues.apache.org/jira/browse/YARN-4462 <https://issues.apache.org/jira/browse/YARN-4462>
> 
> Thanks
> 
> Kai Sasaki
> 
> 
>> On May 15, 2017, at 16:17, wuchang <583424568@qq.com <mailto:583424568@qq.com>>
wrote:
>> 
>> Below is my fair-scheduler.xml file:
>> 
>> <allocations>
>>     <queue name="highPriority">
>>        <minResources>100000 mb, 30 vcores</minResources>
>>        <maxResources>500000 mb, 100 vcores</maxResources>
>>        <weight>0.35</weight>
>>        <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
>>        <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
>>        <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
>>     </queue>
>>     <queue name="default">
>>        <minResources>25000 mb, 20 vcores</minResources>
>>        <maxResources>225000 mb, 70 vcores</maxResources>
>>        <weight>0.14</weight>
>>        <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
>>        <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
>>        <fairSharePreemptionThreshold>0.5</fairSharePreemptionThreshold>
>>        <maxAMShare>-1.0f</maxAMShare>
>>     </queue>
>>     <queue name="ep">
>>        <minResources>100000 mb, 30 vcores</minResources>
>>        <maxResources>600000 mb, 100 vcores</maxResources>
>>        <weight>0.42</weight>
>>        <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
>>        <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
>>        <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
>>        <maxAMShare>-1.0f</maxAMShare>
>>     </queue>
>>     <queue name="vip">
>>        <minResources>15000 mb, 20 vcores</minResources>
>>        <maxResources>120000 mb, 30 vcores</maxResources>
>>        <weight>0.09</weight>
>>        <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
>>        <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
>>        <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
>>        <maxAMShare>-1.0f</maxAMShare>
>>      </queue>
>> </allocations>
>> 
>> Since the root.ep queue and root.highPriority queue is so important that anytime
I don’t want them to be preempted.I know that if the resource of them is preempted , their
container maybe killed and thus it may take much more time for applications running on them
to finish.
>> So , Any solutions? 
> 


Mime
View raw message