Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 33FE118214 for ; Thu, 16 Jul 2015 08:00:14 +0000 (UTC) Received: (qmail 74945 invoked by uid 500); 16 Jul 2015 08:00:07 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 74824 invoked by uid 500); 16 Jul 2015 08:00:07 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 74811 invoked by uid 99); 16 Jul 2015 08:00:06 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2015 08:00:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 628A31827F2 for ; Thu, 16 Jul 2015 08:00:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.176 X-Spam-Level: **** X-Spam-Status: No, score=4.176 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, MISSING_HEADERS=1.207, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id iv4ruyXqMmsH for ; Thu, 16 Jul 2015 08:00:04 +0000 (UTC) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [119.145.14.65]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 614D220CF5 for ; Thu, 16 Jul 2015 07:59:44 +0000 (UTC) Received: from 172.24.2.119 (EHLO szxeml434-hub.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id COU86553; Thu, 16 Jul 2015 15:53:12 +0800 (CST) Received: from SZXEML505-MBX.china.huawei.com ([169.254.1.166]) by szxeml434-hub.china.huawei.com ([10.82.67.225]) with mapi id 14.03.0158.001; Thu, 16 Jul 2015 15:53:09 +0800 From: "Naganarasimha G R (Naga)" CC: "user@hadoop.apache.org" Subject: FW: Is it valid to use userLimit in calculating maxApplicationsPerUser ? Thread-Topic: Is it valid to use userLimit in calculating maxApplicationsPerUser ? Thread-Index: AdC/mcvFaxMgnqm9R12nzRa38Mc39wAAo7oi Date: Thu, 16 Jul 2015 07:53:08 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.212.45.80] Content-Type: multipart/alternative; boundary="_000_AD354F56741A1B47882A625909A59C692BE0218ASZXEML505MBXchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected --_000_AD354F56741A1B47882A625909A59C692BE0218ASZXEML505MBXchi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Folks , Came across one scenario where in maxApplications @ cluster level(2 node) w= as set to a low value like 10 and based on capacity configuration for a par= ticular queue it was coming to 2 as value, but further while calculating ma= xApplicationsPerUser formula used is : maxApplicationsPerUser =3D (int)(maxApplications * (userLimit / 100.0f) * u= serLimitFactor); but the definition of userLimit in the documentation is : Each queue enforces a limit on the percentage of resources allocated to a u= ser at any given time, if there is demand for resources. The user limit can= vary between a minimum and maximum value. The the former (the minimum valu= e) is set to this property value and the latter (the maximum value) depends= on the number of users who have submitted applications. For e.g., suppose = the value of this property is 25. If two users have submitted applications = to a queue, no single user can use more than 50% of the queue resources. If= a third user submits an application, no single user can use more than 33% = of the queue resources. With 4 or more users, no user can use more than 25%= of the queues resources. A value of 100 implies no user limits are imposed= . The default is 100. Value is specified as a integer. So was wondering how a minimum limit is made used in a formula to calculate= max applications for a user, suppose i set "yarn.scheduler.capacity..minimum-user-limit-percent" to 20 assuming at least 20% of queue at= the minimum is available for a queue but based on the formula maxApplicati= onsPerUser is getting set to zero. According to the definition of the property max is based on the current no.= of active users, so i feel this formula is wrong. P.S. userLimitFactor was configured as default 1, but what i am wondering i= s whether its valid to use it in combination with userlimit to find max app= s per user. Please correct me if my understanding is wrong? if its a bug would like to = raise and solve it . + Naga --_000_AD354F56741A1B47882A625909A59C692BE0218ASZXEML505MBXchi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi Folks ,
Came across one s= cenario where in maxApplications @ cluster level(2 node) was set to a = low value like 10 and based on capacity configuration for a particular queue it was coming to 2 as value, but further while calc= ulating maxApplicationsPerUser formula used is : 

max= ApplicationsPerUser =3D (int)(maxApplic= ations * (userLimit / 100.0f) * userLimitFactor);

but the definition of u= serLimit  in the docu= mentation is :
Each qu= eue enforces a limit on the percentage of resources allocated to a user at = any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value= . = The the former (the minimum value) is set to this property value and the latter (the maximum value) depends o= n the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two u= sers have submitted applications to a queue, no single user can use more th= an 50% of the queue resources. If a third user submits an application, no s= ingle user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25= % of the queues resources. A value of 100 implies no user limits are impose= d. The default is 100. Value is specified as a integer.

So was wondering how a minimum limit is made used in a formula to c= alculate max applications for a user, suppose i set "yarn.scheduler.capacity.<queue-path>.minimum-= user-limit-percent" to 20  assuming at = least 20% of queue at the minimum is available for a queue but based on the= formula maxApplicati= onsPerUser  is getting set to zero. 
According to the definition of the property max is based on th= e current no. of active users, so i feel this formula&n= bsp;is wrong.
P.S. userLimitFactor was config= ured as default 1, but what i am wondering is whether its valid to use it i= n combination with userlimit to find max apps per user.

Please correct me if my understanding i= s wrong? if its a bug would like to raise and solve it .

+ Naga
--_000_AD354F56741A1B47882A625909A59C692BE0218ASZXEML505MBXchi_--