hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3508) Preemption processing occuring on the main RM dispatcher
Date Tue, 30 Jun 2015 18:23:27 GMT

    [ https://issues.apache.org/jira/browse/YARN-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608797#comment-14608797
] 

Wangda Tan commented on YARN-3508:
----------------------------------

I think my comment confusing you: https://issues.apache.org/jira/browse/YARN-3508?focusedCommentId=14606576&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14606576.
I was thinking yesterday that we need to directly send preemption events to scheduler event
dispatcher. But if we expose scheduler event dispatcher in RMContext, more event dispatcher
will be exposed in the future, that is against the reason why we have a central RM dispatcher.

You patch basically looks good, some minor comments:
- Instead of passing EventHandler<SchedulerEvent> to Proportional.., you should pass
RMContext.
- Not caused by your patch, save e.getKey() to ApplicationAttemptId in containerBasedPreemptOrKill
avoid multiple getKey
- Source: ProportionalCapacityPreemptionPolicy, should be {{SchedulingEditPolicy}}

Other looks good to me.

> Preemption processing occuring on the main RM dispatcher
> --------------------------------------------------------
>
>                 Key: YARN-3508
>                 URL: https://issues.apache.org/jira/browse/YARN-3508
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Varun Saxena
>         Attachments: YARN-3508.002.patch, YARN-3508.01.patch, YARN-3508.03.patch, YARN-3508.04.patch
>
>
> We recently saw the RM for a large cluster lag far behind on the AsyncDispacher event
queue.  The AsyncDispatcher thread was consistently blocked on the highly-contended CapacityScheduler
lock trying to dispatch preemption-related events for RMContainerPreemptEventDispatcher. 
Preemption processing should occur on the scheduler event dispatcher thread or a separate
thread to avoid delaying the processing of other events in the primary dispatcher queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message