hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Radwan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
Date Tue, 09 Oct 2012 17:14:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472551#comment-13472551

Ahmed Radwan commented on YARN-140:

Thanks Tucu,

Yes Tucu, I already had it like that in rev4 of the patch, but after a second thought I changed
it in the latest patch. It is true that the capacity of root should be always 100, or otherwise
an exception will be thrown, see ParentQueue.java:116:

    if (rootQueue &&
        (rawCapacity != CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE)) {
      throw new IllegalArgumentException("Illegal " +
          "capacity of " + rawCapacity + " for queue " + queueName +
          ". Must be " + CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE);

However, I think we shouldn't overwrite a configuration given by the user, we can just provide
the correct default if it is missing. 

Overwriting a configuration can have a negative consequence of having a configuration file
contents which contradicts what is really used by the scheduler, which is problematic.

Using the latest patch, if the user sets the root capacity to a wrong value, then the system
will throw an Exception, and the user will know it is wrong and correct it. But if the user
misses it, the default will be picked. This is why I changed to {{&&}}. What do you
> Add capacity-scheduler-default.xml to provide a default set of configurations for the
capacity scheduler.
> ---------------------------------------------------------------------------------------------------------
>                 Key: YARN-140
>                 URL: https://issues.apache.org/jira/browse/YARN-140
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>            Reporter: Ahmed Radwan
>            Assignee: Ahmed Radwan
>         Attachments: YARN-140.patch, YARN-140_rev2.patch, YARN-140_rev3.patch, YARN-140_rev4.patch,
YARN-140_rev5_onlyForJenkins.patch, YARN-140_rev5.patch, YARN-140_rev5_svn_mv.patch
> When setting up the capacity scheduler users are faced with problems like:
> {code}
> FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
> java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
> {code}
> Which basically arises from missing basic configurations, which in many cases, there
is no need to explicitly provide, and a default configuration will be sufficient. For example,
to address the error above, the user need to add a capacity of 100 to the root queue.
> So, we need to add a capacity-scheduler-default.xml, this will be helpful to provide
the basic set of default configurations required to run the capacity scheduler. The user can
still override existing configurations or provide new ones in capacity-scheduler.xml. This
is similar to *-default.xml vs *-site.xml for yarn, core, mapred, hdfs, etc.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message