hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
Date Wed, 27 Feb 2013 21:53:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588818#comment-13588818
] 

Bikas Saha commented on YARN-365:
---------------------------------

Do we need to worry about there being overlap between the 2 lists. i.e. a newlyLaunchedContainer
also got completed by the time the slow RM handled the NM updates?
{code}
+  private synchronized void nodeUpdate(RMNode nm) {
     if (LOG.isDebugEnabled()) {
       LOG.debug("nodeUpdate: " + nm + " clusterResources: " + clusterResource);
     }
-                  
-    FiCaSchedulerNode node = getNode(nm.getNodeID());
 
+    FiCaSchedulerNode node = getNode(nm.getNodeID());
+    List<UpdatedContainerInfo> containerInfoList = nm.pullContainerUpdates();
+    List<ContainerStatus> newlyLaunchedContainers = new ArrayList<ContainerStatus>();
+    List<ContainerStatus> completedContainers = new ArrayList<ContainerStatus>();
+    for(UpdatedContainerInfo containerInfo : containerInfoList) {
+      newlyLaunchedContainers.addAll(containerInfo.getNewlyLaunchedContainers());
+      completedContainers.addAll(containerInfo.getCompletedContainers());
+    }
+    
{code}
Note than this problem (if it is a problem) exists regardless of this change because a container
may start and complete within the NM heartbeat interval. However, chances of hitting it are
low before this change because the heartbeat interval is short and so the RM never see a node
update in which the same container both launches and completes. After this change, with a
slow RM, this can easily happen, specially because we are simply concatenating both sub-lists.
                
> Each NM heartbeat should not generate an event for the Scheduler
> ----------------------------------------------------------------
>
>                 Key: YARN-365
>                 URL: https://issues.apache.org/jira/browse/YARN-365
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager, scheduler
>    Affects Versions: 0.23.5
>            Reporter: Siddharth Seth
>            Assignee: Xuan Gong
>             Fix For: 2.0.4-beta
>
>         Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch, YARN-365.1.patch,
YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch,
YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch
>
>
> Follow up from YARN-275
> https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message