Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B9CB4200C67 for ; Mon, 15 May 2017 10:24:01 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B834A160BC2; Mon, 15 May 2017 08:24:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AE569160BC1 for ; Mon, 15 May 2017 10:24:00 +0200 (CEST) Received: (qmail 70534 invoked by uid 500); 15 May 2017 08:23:58 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70523 invoked by uid 99); 15 May 2017 08:23:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 May 2017 08:23:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D719E1AF8E8 for ; Mon, 15 May 2017 08:23:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.439 X-Spam-Level: X-Spam-Status: No, score=-0.439 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, RP_MATCHES_RCVD=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_DKIM_INVALID=0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=fail (1024-bit key) reason="fail (body has been altered)" header.d=qq.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id JJpfNKziAIsx for ; Mon, 15 May 2017 08:23:52 +0000 (UTC) Received: from smtpbg65.qq.com (smtpbg65.qq.com [103.7.28.233]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 951155F520 for ; Mon, 15 May 2017 08:23:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1494836616; bh=ps9Rt1LKwEVNn6wNBN6nnXIBjWUuafDulJs9fxzEvcU=; h=From:Message-Id:Content-Type:Mime-Version:Subject:Date:In-Reply-To:Cc:To:References; b=uGb+ZqyTg8UqXilitjGAZa3aBlEK6lYKC644ek8uOpHrWDglhO6H6U2k/GMZuNBfT w95050CZWMXQTikVqR5hN7QedGM0vCb7Jn+Qm2/9yOzQxQnaFlcR3vNn/1qrAjm8sP dvE+XE9BFN03w4P/H6wXC0pcU3cu2tskUWG2Y/Y8= X-QQ-mid: esmtp27t1494836614tqkxx1idu Received: from [10.235.253.108] (unknown [61.135.255.84]) by esmtp4.qq.com (ESMTP) with id ; Mon, 15 May 2017 16:23:33 +0800 (CST) X-QQ-SSF: 00000000000000F0FM301F00000000Z X-QQ-FEAT: /WpsoPXzXg50+GiQkijUAMTcr5hnZHY2Bs2uve9+v+MA17130+wcCv6fPIYWB 2p9KuCFGocCQhxtQk8gSkJEHbGVzN9oS8ERXCIYi0ykQj4v0AGAau4t7M1JJiLdsimrHURy 8fXWDS6yg328m7hVIRbvgJ8R6YoTy5Wsrx4pO1zNN3xPbrJbM8VFT7nwyYqh0HfQHEAW8ro 5KQWz6HBl43tUBKN5/HkIimRybOHUtBvC0zn0Ra2t99UPbtnAKhwcqu+ODof+pZ3xm1m9RY 6YhQRYyEaYUHdEUCD+ZcqNf+2kF7utB7y6YuFjpvwBavqgQpMsilS9x00= X-QQ-GoodBg: 0 From: wuchang <583424568@qq.com> Message-Id: <138CDD0A-617D-45B8-BBD1-3D8A74E97691@qq.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_D7CB02D2-D767-45B3-A035-04E0EC373DBE" Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: How to make a queue cannot be preempted under FairScheduler? Date: Mon, 15 May 2017 16:23:33 +0800 In-Reply-To: <1D19DDD3-E018-4FBC-B4EF-234FCB92D3BE@me.com> Cc: user@hadoop.apache.org To: Sasaki Kai References: <30237817-5D6C-4FD6-8BC1-2DA688A0232C@qq.com> <1D19DDD3-E018-4FBC-B4EF-234FCB92D3BE@me.com> X-Mailer: Apple Mail (2.3273) X-QQ-SENDSIZE: 520 Feedback-ID: esmtp:qq.com:bgforeign:bgforeign1 X-QQ-Bgrelay: 1 archived-at: Mon, 15 May 2017 08:24:01 -0000 --Apple-Mail=_D7CB02D2-D767-45B3-A035-04E0EC373DBE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Great thanks.=20 It seems that for my Hadoop 2.7.3 it=E2=80=99s absolutely impossible to = prevent a queue from being preempted.But I have tried to make a queue = which is more important a higher fairSharePreemptionThreshold value , = and a queue which is less important a lower = fairSharePreemptionThreshold value , I believe this will make the = queue which is more important to be easier to preempt the resources of = queue which is less important, thus the resource of important queue is = harder to be preempted by other unimportant queues; I don=E2=80=99 t know if this is just a workaround. > On May 15, 2017, at 16:08, Sasaki Kai = wrote: >=20 > Hi >=20 > I believe there is no nice way to prevent preemption from specific = queue.=20 > Only thing we can do is disabling preemption completely. >=20 > > yarn.scheduler.fair.preemption > true > >=20 > =46rom Hadoop 2.9.0 or 3.0.0-alpha1 we will be able to use = allowPreemption=46rom property to forbid preemption from the queue. >=20 > > false > >=20 > https://issues.apache.org/jira/browse/YARN-4462 = >=20 > Thanks >=20 > Kai Sasaki >=20 >=20 >> On May 15, 2017, at 16:17, wuchang <583424568@qq.com = > wrote: >>=20 >> Below is my fair-scheduler.xml file: >>=20 >> >> >> 100000 mb, 30 vcores >> 500000 mb, 100 vcores >> 0.35 >> 20 >> 25 >> = 0.8 >> >> >> 25000 mb, 20 vcores >> 225000 mb, 70 vcores >> 0.14 >> 20 >> 25 >> = 0.5 >> -1.0f >> >> >> 100000 mb, 30 vcores >> 600000 mb, 100 vcores >> 0.42 >> 20 >> 25 >> = 0.8 >> -1.0f >> >> >> 15000 mb, 20 vcores >> 120000 mb, 30 vcores >> 0.09 >> 20 >> 25 >> = 0.8 >> -1.0f >> >> >>=20 >> Since the root.ep queue and root.highPriority queue is so important = that anytime I don=E2=80=99t want them to be preempted.I know that if = the resource of them is preempted , their container maybe killed and = thus it may take much more time for applications running on them to = finish. >> So , Any solutions?=20 >=20 --Apple-Mail=_D7CB02D2-D767-45B3-A035-04E0EC373DBE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Great thanks. 
It seems that for my = Hadoop 2.7.3 it=E2=80=99s absolutely  impossible to prevent a queue = from being preempted.But I have tried to make a queue which is more = important a higher fairSharePreemptionThreshold value , and a queue = which is less important a lower  fairSharePreemptionThreshold =  value  , I believe this will make the queue which is more = important to be easier to preempt the resources of queue which is less = important, thus the resource of important queue is harder to be = preempted by other unimportant queues;

I don=E2=80=99 t know if this is just a = workaround.

On = May 15, 2017, at 16:08, Sasaki Kai <sasaki@treasure-data.com> wrote:

Hi

I believe there is no = nice way to prevent preemption from specific queue. 
Only thing we can do is disabling preemption = completely.

<property> <name>yarn.scheduler.fair.preemption</name> <value>true</value> </property>

=46rom Hadoop 2.9.0 or 3.0.0-alpha1 we will be able to use = allowPreemption=46rom property to forbid preemption from the = queue.

<queue name=3D=E2=80=9ChighPriority">
    <allowPreemptionFrom>false</allowPreemptionFrom>
</queue>

https://issues.apache.org/jira/browse/YARN-4462

Thanks

Kai Sasaki


On May 15, 2017, at 16:17, wuchang <583424568@qq.com> = wrote:

Below is my = fair-scheduler.xml file:

<allocations>
 =   <queue name=3D"highPriority">
  =      <minResources>100000 mb, 30 = vcores</minResources>
      =  <maxResources>500000 mb, 100 = vcores</maxResources>
      =  <weight>0.35</weight>
  =     =  <minSharePreemptionTimeout>20</minSharePreemptionTimeout>= ;
      =  <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout&= gt;
      =  <fairSharePreemptionThreshold>0.8</fairSharePreemptionThres= hold>
    </queue>
    <queue name=3D"default">
       <minResources>25000 mb, 20 = vcores</minResources>
      =  <maxResources>225000 mb, 70 = vcores</maxResources>
      =  <weight>0.14</weight>
  =     =  <minSharePreemptionTimeout>20</minSharePreemptionTimeout>= ;
      =  <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout&= gt;
      =  <fairSharePreemptionThreshold>0.5</fairSharePreemptionThres= hold>
      =  <maxAMShare>-1.0f</maxAMShare>
    </queue>
  =   <queue name=3D"ep">
    =    <minResources>100000 mb, 30 = vcores</minResources>
      =  <maxResources>600000 mb, 100 = vcores</maxResources>
      =  <weight>0.42</weight>
  =     =  <minSharePreemptionTimeout>20</minSharePreemptionTimeout>= ;
      =  <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout&= gt;
      =  <fairSharePreemptionThreshold>0.8</fairSharePreemptionThres= hold>
      =  <maxAMShare>-1.0f</maxAMShare>
    </queue>
  =   <queue name=3D"vip">
    =    <minResources>15000 mb, 20 = vcores</minResources>
      =  <maxResources>120000 mb, 30 = vcores</maxResources>
      =  <weight>0.09</weight>
  =     =  <minSharePreemptionTimeout>20</minSharePreemptionTimeout>= ;
      =  <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout&= gt;
      =  <fairSharePreemptionThreshold>0.8</fairSharePreemptionThres= hold>
      =  <maxAMShare>-1.0f</maxAMShare>
     </queue>
</allocations>

Since the root.ep = queue and root.highPriority queue is so important that = anytime I don=E2=80=99t want them to be preempted.I know that if the = resource of them is preempted , their container maybe killed and thus it = may take much more time for applications running on them to = finish.
So , Any = solutions? 


= --Apple-Mail=_D7CB02D2-D767-45B3-A035-04E0EC373DBE--