hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5605) Preempt containers (all on one node) to meet the requirement of starved applications
Date Thu, 08 Sep 2016 00:26:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472246#comment-15472246
] 

ASF GitHub Bot commented on YARN-5605:
--------------------------------------

Github user kambatla commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/124#discussion_r77928672
  
    --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
    @@ -223,17 +225,76 @@ public void setPolicy(SchedulingPolicy policy)
         }
         super.policy = policy;
       }
    -  
    +
       @Override
    -  public void recomputeShares() {
    +  public void updateInternal(boolean checkStarvation) {
         readLock.lock();
         try {
           policy.computeShares(runnableApps, getFairShare());
    +      if (checkStarvation) {
    +        identifyStarvedApplications();
    +      }
         } finally {
           readLock.unlock();
         }
       }
     
    +  /**
    +   * Helper method to identify starved applications. This needs to be called
    +   * ONLY from {@link #updateInternal}, after the application shares
    +   * are updated.
    +   *
    +   * A queue can be starving due to fairshare or minshare.
    +   *
    +   * Minshare is defined only on the queue and not the applications.
    +   * Fairshare is defined for both the queue and the applications.
    +   *
    +   * If this queue is starved due to minshare, we need to identify the most
    +   * deserving apps if they themselves are not starved due to fairshare.
    +   *
    +   * If this queue is starving due to fairshare, there must be at least
    +   * one application that is starved. And, even if the queue is not
    +   * starved due to fairshare, there might still be starved applications.
    +   */
    +  private void identifyStarvedApplications() {
    +    // First identify starved applications and track total amount of
    +    // starvation (in resources)
    +    Resource fairShareStarvation = Resources.clone(none());
    +    TreeSet<FSAppAttempt> appsWithDemand = fetchAppsWithDemand();
    +    for (FSAppAttempt app : appsWithDemand) {
    +      Resource appStarvation = app.fairShareStarvation();
    +      if (Resources.equals(Resources.none(), appStarvation))  {
    +        break;
    +      } else {
    --- End diff --
    
    The else is required. We are iterating through all apps with unmet demand. Since the list
is sorted by fairshare starvation, we could stop iterating when we hit an app that is at or
above its fairshare. Added a comment to clarify that. 


> Preempt containers (all on one node) to meet the requirement of starved applications
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-5605
>                 URL: https://issues.apache.org/jira/browse/YARN-5605
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: yarn-5605-1.patch
>
>
> Required items:
> # Identify starved applications
> # Identify a node that has enough containers from applications over their fairshare.
> # Preempt those containers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message