hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5262) Allow specifying min shares as percentage of cluster
Date Sun, 15 Feb 2009 00:47:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673558#action_12673558

Matei Zaharia commented on HADOOP-5262:

I suggested we allow both modes because we have been using the fixed-number-of-slots model
at Facebook. We mostly use min share to guarantee that ETL jobs and other jobs with deadlines
get enough slots to finish in time. Thus it makes more sense to set the min share to something
small but just enough to be sure that the job will finish, and this is easier to think of
in terms of slots than percents. I understand that the Yahoo use case is different and requires
assigning percent shares to organizations.

> Allow specifying min shares as percentage of cluster
> ----------------------------------------------------
>                 Key: HADOOP-5262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5262
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Priority: Minor
> Currently the guaranteed shares for pools in the fair scheduler are specified as a number
of slots. For organizations where a group pays X% of the cluster and the actual number of
nodes in the cluster varies due to failures, expansion, etc over time, it would be useful
to support a guaranteed share given as a percentage too. This would just let you write in
the config file something like <minMaps>5%</minMaps> instead of <minMaps>42</minMaps>.
The scheduler would need to recompute what this means in terms of number of slots on every
update (probably through some kind of update(ClusterStatus) method in PoolManager).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message