hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zian Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6124) Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues
Date Mon, 20 Nov 2017 22:49:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259975#comment-16259975

Zian Chen commented on YARN-6124:

Thank you [~eepayne] for your comments. for AdminService#refreshQueues, the first comment,
I totally agree with your suggestion since it calls all of the schedulers, not just capacity
scheduler. I'll update the comments according to your suggestions.

For the reason why conf.size() is been called in AdminService#refreshQueues, the reason exactly
comes from the exception which you have observed in your newest comment. Let me explain it
in details,

I agree with your comments that SchedulingMonitorManager#updateSchedulingMonitors will call
Configuration#getBoolean which will call Configuration#getProps. However,  if we look into
the logic inside CapacityScheduler#reinitialize after applying YARN-6124.3.patch, we need
to make sure we change Queue configuration without throwing any exception then we can enable
preemption related configuration by calling super.reinitialize(which will call schedulingMonitorManager#updateSchedulingMonitors
in the end), but when we change the Queue configuration, we will call csConfProvider#loadConfiguration,
however at this point, if we don't call Configuration#getProps by any chance, the configuration
will not really load into memory, meanwhile the input stream we specified for reading the
Configuration inside AdminService#refreshQueues is already closed, which leads to the refreshQueues:
com.ctc.wstx.exc.WstxIOException: Stream closed exception occurs. 

Because of this situation, we need to invoke Configuration#getProps right after we call Configuration#getConfiguration
in order to load all the properties we need before the input stream close. Wangda and I also
think there should be a better way to achieve this, so we went through all the method which
invokes Configuration#getProps but we only find conf.size() which can be used for our purpose.
That's why conf.size() is been called here. 

Hope this can explain the reason we call conf.size() here, we'll appreciate your help if you
could find a better way to avoid the WstxIOException and load the configuration. Thank you
so much for your help!

> Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues
> -----------------------------------------------------------------------------------------
>                 Key: YARN-6124
>                 URL: https://issues.apache.org/jira/browse/YARN-6124
>             Project: Hadoop YARN
>          Issue Type: Task
>            Reporter: Wangda Tan
>            Assignee: Zian Chen
>         Attachments: YARN-6124.wip.1.patch, YARN-6124.wip.2.patch, YARN-6124.wip.3.patch
> Now enabled / disable / update SchedulingEditPolicy config requires restart RM. This
is inconvenient when admin wants to make changes to SchedulingEditPolicies.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message