hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation
Date Wed, 10 Dec 2014 17:02:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241381#comment-14241381
] 

Junping Du commented on YARN-2637:
----------------------------------

Thanks [~cwelch] for updating the patch! The patch looks good in overall, some minor comments:
bq.
{code}
+  public int getMaximumActiveApplicationsForQueue(String queue) {
+    int maxActiveApplicationsForQueue = 
+      getInt(getQueuePrefix(queue) + MAXIMUM_ACTIVE_APPLICATIONS_SUFFIX, 
+        getInt(DEFAULT_MAXIMUM_ACTIVE_QUEUE_APPLICATIONS, -1));
+    return maxActiveApplicationsForQueue;
+  }
{code}
If my understanding is correct, here we are trying to get a per queue value first, if not
set, we are trying to get a value that is common for each queue, and get -1 at last which
means we will try to calculate this value later. 
Do we set any default value for DEFAULT_MAXIMUM_ACTIVE_QUEUE_APPLICATIONS somewhere (in code
or capacity-scheduler.xml)? I think the answer is no, so we better to document it somewhere
so that user can understand what to do.

bq. 
{code}
+    if (maxActiveApplicationsForQueue != -1) {
+      //is manually configured
+      maxActiveApplications = maxActiveApplicationsForQueue;
+    } else {
+      maxActiveApplications =
{code}
Do we need to check the value valid for maxActiveApplicationsForQueue? If user set some another
minus value, better to throw some warning message here.

{code}
+	  
+	  //Verify the value for getAMResourceLimit for queues with < .1 maxcap
+	  Resource clusterResource = Resource.newInstance(50 * GB, 50);
+	  
+	  a.updateClusterResource(clusterResource);
+    assertEquals(Resources.multiply(clusterResource, 
+      a.getAbsoluteMaximumCapacity() * a.getMaxAMResourcePerQueuePercent()), 
+      a.getAMResourceLimit());
+	  
+	  b.updateClusterResource(clusterResource);
+    assertEquals(Resources.multiply(clusterResource, 
+      b.getAbsoluteMaximumCapacity() * b.getMaxAMResourcePerQueuePercent()), 
+      b.getAMResourceLimit());
{code}
The format should be adjusted and TAB space should be replaced with white space.

Other looks fine to me. [~leftnoteasy], do you have additional comments?

> maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-2637
>                 URL: https://issues.apache.org/jira/browse/YARN-2637
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Wangda Tan
>            Assignee: Craig Welch
>            Priority: Critical
>         Attachments: YARN-2637.0.patch, YARN-2637.1.patch, YARN-2637.12.patch, YARN-2637.13.patch,
YARN-2637.15.patch, YARN-2637.16.patch, YARN-2637.2.patch, YARN-2637.6.patch, YARN-2637.7.patch,
YARN-2637.9.patch
>
>
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be activated in following
way:
> {code}
>     for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); 
>          i.hasNext(); ) {
>       FiCaSchedulerApp application = i.next();
>       
>       // Check queue limit
>       if (getNumActiveApplications() >= getMaximumActiveApplications()) {
>         break;
>       }
>       
>       // Check user limit
>       User user = getUser(application.getUser());
>       if (user.getActiveApplications() < getMaximumActiveApplicationsPerUser()) {
>         user.activateApplication();
>         activeApplications.add(application);
>         i.remove();
>         LOG.info("Application " + application.getApplicationId() +
>             " from user: " + application.getUser() + 
>             " activated in queue: " + getQueueName());
>       }
>     }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent  = 0.2, the maximum resource that
AM can use is 200M, assuming minimum_allocation=1M, #am can be launched is 200, and if user
uses 5M for each AM (> minimum_allocation). All apps can still be activated, and it will
occupy all resource of a queue instead of only a max_am_resource_percent of a queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message