Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 14 Feb 2017 22:34:41 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13041452.1486580325000.81071.1487111681780@Atlassian.JIRA>
In-Reply-To: <JIRA.13041452.1486580325000@Atlassian.JIRA>
References: <JIRA.13041452.1486580325000@Atlassian.JIRA> <JIRA.13041452.1486580325598@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6163) FS Preemption is a trickle for
 severely starved applications
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 14 Feb 2017 22:34:47 -0000


    [ https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866841#comment-15866841 ] 

ASF GitHub Bot commented on YARN-6163:
--------------------------------------

Github user templedf commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/192#discussion_r101159725
  
    --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java ---
    @@ -1147,24 +1147,32 @@ private static boolean checkAndMarkRRVisited(
        * starvation.
        */
       List<ResourceRequest> getStarvedResourceRequests() {
    +    // List of RRs we build in this method to return
         List<ResourceRequest> ret = new ArrayList<>();
    +
    +    // Track visited RRs to avoid the same RR at multiple locality levels
         Map<Priority, List<Resource>> visitedRRs= new HashMap<>();
     
    +    // Start with current starvation and track the pending amount
         Resource pending = getStarvation();
         for (ResourceRequest rr : appSchedulingInfo.getAllResourceRequests()) {
           if (Resources.isNone(pending)) {
    +        // Found enough RRs to match the starvation
             break;
           }
    +
    +      // See if we have already seen this RR
           if (checkAndMarkRRVisited(visitedRRs, rr)) {
             continue;
           }
     
    -      // Compute the number of containers of this capability that fit in the
    -      // pending amount
    +      // A RR can have multiple containers of a capability. We need to
    +      // compute the number of containers that fit in "pending".
           int ratio = (int) Math.floor(
    --- End diff --
    
    Given that ratio is the number of containers that fit in "pending," ratio is probably a bad name.  That was a good chunk of my initial confusion.


> FS Preemption is a trickle for severely starved applications
> ------------------------------------------------------------
>
>                 Key: YARN-6163
>                 URL: https://issues.apache.org/jira/browse/YARN-6163
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>    Affects Versions: 2.9.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an application starved. This marking happens only on the update call that runs every 500ms.  Due to this, an application that is severely starved takes forever to reach fairshare based on preemptions.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org