hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
Date Mon, 13 Mar 2017 23:00:43 GMT

    [ https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923141#comment-15923141
] 

ASF GitHub Bot commented on YARN-5829:
--------------------------------------

Github user kambatla commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/201#discussion_r105790122
  
    --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
    @@ -118,23 +178,56 @@ synchronized FSAppAttempt getReservedAppSchedulable() {
        *
        * @param containers container to mark
        */
    -  void addContainersForPreemption(Collection<RMContainer> containers) {
    -    containersForPreemption.addAll(containers);
    +  void addContainersForPreemption(Collection<RMContainer> containers,
    +                                  FSAppAttempt appAttempt) {
    +    for(RMContainer container : containers) {
    +      containersForPreemption.put(container, appAttempt);
    +    }
       }
     
       /**
        * @return set of containers marked for preemption.
        */
       Set<RMContainer> getContainersForPreemption() {
    -    return containersForPreemption;
    +    return containersForPreemption.keySet();
       }
     
       /**
        * Remove container from the set of containers marked for preemption.
    +   * Reserve the preempted resources for the app that requested
    +   * the preemption.
        *
        * @param container container to remove
        */
    -  void removeContainerForPreemption(RMContainer container) {
    +  private synchronized void removeContainerForPreemption(RMContainer container) {
    +    FSAppAttempt app = containersForPreemption.get(container);
    +    if (app != null) {
    +      Resource containerSize =
    +          Resources.clone(container.getAllocatedResource());
    +      if (!reservedApp.containsKey(app)) {
    +        reservedApp.put(app, containerSize);
    +      } else {
    +        Resources.addTo(reservedApp.get(app),
    +            Resources.clone(containerSize));
    +      }
    +    }
         containersForPreemption.remove(container);
       }
    +
    +  /**
    +   * Release an allocated container on this node.
    +   * It also releases from the reservation list to trigger preemption
    +   * allocations.
    +   * @param containerId ID of container to be released.
    +   * @param releasedByNode whether the release originates from a node update.
    +   */
    +  @Override
    +  public synchronized void releaseContainer(ContainerId containerId,
    +                                            boolean releasedByNode) {
    +    RMContainer container = getContainer(containerId);
    +    super.releaseContainer(containerId, releasedByNode);
    +    if (container != null) {
    +      removeContainerForPreemption(container);
    --- End diff --
    
    This is nice to see. We should have done this to begin with. 


> FS preemption should reserve a node before considering containers on it for preemption
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5829
>                 URL: https://issues.apache.org/jira/browse/YARN-5829
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>            Reporter: Karthik Kambatla
>            Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts identified containers.
If this node is not reserved for a specific application, any other application could be allocated
resources on this node. 
> Reserving the node for the starved application before preempting containers would help
avoid this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message