hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5262) Allow specifying min shares as percentage of cluster
Date Mon, 16 Feb 2009 19:22:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674038#action_12674038
] 

Matei Zaharia commented on HADOOP-5262:
---------------------------------------

I think the second config variable makes sense. When we describe it in the documentation,
we can say "either you configure the pool to a percent of total cluster capacity, or specify
an exact number of maps and reduces". Certainly nobody pays for 10% of reduces but 20% of
maps.

The main difficulty I see with this and fair scheduler configuration in general is how to
provide feedback to the operator. Right now, if the fair scheduler detects something wrong
in the config file at startup, it throws a RuntimeException to prevent the JobTracker from
starting, thus bringing the problem to the operator's attention. If it detects a problem when
reloading a config file at runtime, it logs it as an ERROR in the JobTracker's log4j log but
it keeps the old settings of the config variables so that the cluster can continue operating.
Because most of the config info is visible on the web UI, the idea is that an operator will
look at the web UI to see whether their change went through. There are several possible enhancements
to improve this:
* Rather than making the config reloading something passive, maybe we can require the operator
to say "reload the config file now" through either a command-like invocation or a button on
the web UI. In response to this, we can display any errors.
* If things don't make sense at runtuime, e.g. total min shares add up to more than 100% because
nodes went away, maybe we can display a warning on the web UI or email the operator (?). (By
the way, right now, if min shares exceed 100%, the scheduler doesn't crash or behave weirdly;
it just treats them as weights and normalizes them to 100%; so it's not the end of the world,
but the operator should know about it.)

> Allow specifying min shares as percentage of cluster
> ----------------------------------------------------
>
>                 Key: HADOOP-5262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5262
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Priority: Minor
>
> Currently the guaranteed shares for pools in the fair scheduler are specified as a number
of slots. For organizations where a group pays X% of the cluster and the actual number of
nodes in the cluster varies due to failures, expansion, etc over time, it would be useful
to support a guaranteed share given as a percentage too. This would just let you write in
the config file something like <minMaps>5%</minMaps> instead of <minMaps>42</minMaps>.
The scheduler would need to recompute what this means in terms of number of slots on every
update (probably through some kind of update(ClusterStatus) method in PoolManager).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message