[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573244#comment-13573244 ] 

Siddharth Seth commented on YARN-365:
-------------------------------------

This isn't very different from configuring all nodes to have a higher heartbeat interval. With a high heartbeat interval, the NM would send a batch of updates over to the RM, and this heartbeat would trigger a scheduling pass.

This change de-links RM scheduling passes from NM heartbeats. The NM can continue to provide node updates with a smaller interval, and the RM handles these, along with a scheduling pass, as and when it chooses to. In this particular case, the scheduler queue ends up with a single scheduling event per node - but will attempt a scheduling run only on the next heartbeat from that node. At a later point, the scheduling could be changed to be triggered by the arrival of a new application - or to just run in a tight loop.

If the scheduler cannot keep up, it ends up scheduling as fast as it can - without node heartbeats affecting the queue size. Also, completed container information from heartbeats is processed earlier (instead of waiting for the event in the queue to be processed) - making each scheduler pass more efficient.

bq. I can see cases where the all at once is actually worse as it will spend more time on a single heartbeat and potentially not get to other things in the queue like apps added as fast. 
The event should not be delayed more than the time required to complete one scheduling pass across all nodes. I don't think this will be much better in the case of a growing scheduler queue.

bq. The only way I can see this being beneficial is if we can aggregate the heartbeats and have the scheduler process less.
Do you mean somehow aggregating heartbeats across nodes ? This approach does aggregate heartbeats for a single node.
                
> Each NM heartbeat should not generate and event for the Scheduler
> -----------------------------------------------------------------
>
>                 Key: YARN-365
>                 URL: https://issues.apache.org/jira/browse/YARN-365
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager, scheduler
>    Affects Versions: 0.23.5
>            Reporter: Siddharth Seth
>            Assignee: Xuan Gong
>         Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch
>
>
> Follow up from YARN-275
> https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira