hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Radwan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
Date Tue, 02 Oct 2012 22:31:08 GMT

    [ https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468128#comment-13468128
] 

Ahmed Radwan commented on YARN-140:
-----------------------------------

Thanks Arun,

Let carify my understanding of the issue a little bit:

- We already have the capacity-scheduler.xml (which contains the default confs) packaged in
the hadoop-yarn-server-resourcemanager.<VERSION>.jar, so it is available in the classpath,
and when the user sets the scheduler class to CS in yarn-site.xml, everything works fine and
the default configutrations in capacity-scheduler.xml are picked and used.

The *problem* happens, when the user tries to provide his own capacity-scheduler.xml and places
it in the conf directory. Since it has the same name as the one in the jar, then it replaces
the one in the classpath and since CapacitySchedulerConfiguration constructor loads a single
resource (capacity-scheduler.xml), then the loaded configuration will be only the ones from
the user's capacity-scheduler.xml, and no defaults are loaded from the file packaged in the
jar.

by default confs, I mean stuff like:

{code}
  <property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
  </property>
{code}

which are all lost and the user need to explicitly add them again to his xml, or otherwise
face errors like the one I mentioned above in the description.

So to address this problem, my attached patch modified the CapacitySchedulerConfiguration
constructor to load two resources instead of one, so the default confs can be preserved and
optionally overridden by the user confs.

One other option is to provide a template conf with our distribution and populate this file
with all the defaults (basically a copy of the one in the package), and then instruct users
to use this file, but I think this option is less convenient as it requires manual preservation
of these defaults by the user and knoweledge of what to change vs what to keep, etc.
                
> Add capacity-scheduler-default.xml to provide a default set of configurations for the
capacity scheduler.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-140
>                 URL: https://issues.apache.org/jira/browse/YARN-140
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>            Reporter: Ahmed Radwan
>            Assignee: Ahmed Radwan
>         Attachments: YARN-140.patch
>
>
> When setting up the capacity scheduler users are faced with problems like:
> {code}
> FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
> java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
> {code}
> Which basically arises from missing basic configurations, which in many cases, there
is no need to explicitly provide, and a default configuration will be sufficient. For example,
to address the error above, the user need to add a capacity of 100 to the root queue.
> So, we need to add a capacity-scheduler-default.xml, this will be helpful to provide
the basic set of default configurations required to run the capacity scheduler. The user can
still override existing configurations or provide new ones in capacity-scheduler.xml. This
is similar to *-default.xml vs *-site.xml for yarn, core, mapred, hdfs, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message