hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rahul k singh (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1105) CapacityScheduler: It should be possible to set queue hard-limit beyond it's actual capacity
Date Mon, 19 Oct 2009 04:43:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767210#action_12767210
] 

rahul k singh commented on MAPREDUCE-1105:
------------------------------------------

Summary for yahoo distribution patch.
==============================
- Remove the existing "mapred.capacity-scheduler.queue.<queue-name>.max.map.slots" and
"mapred.capacity-scheduler.queue.<queue-name>.max.reduce.slots" variables , these where
used to throttle the queue, i.e, these were the hard limit and not allowing
queue to grow further.

- Added the new parameter "mapred.capacity-scheduler.queue.<queue-name>.maximum-capacity"

maximum-capacity defines a limit beyond which a queue cannot use the capacity of the cluster.
This provides a means to limit how much excess capacity a queue can use. By default, there
is no limit.The maximum-capacity of a queue can only be greater than or equal to its minimum
capacity. Default value of -1 implies a queue can use complete capacity of the cluster.
This property could be to curtail certain jobs which are long running in nature from occupying
more than a certain percentage of the cluster, which in the absence of pre-emption, could
lead to capacity guarantees of other queues being affected. One important thing to note is
that maximum-capacity is a percentage , so based on the cluster's capacity the max capacity
would change. So if large no of nodes or racks get added to the cluster , max Capacity in
absolute terms would increase accordingly.

- Added some testcases for unit testing the maximum-capacity knob.

- remove testcase for max.map.slots and max.reduce.slots.

Summary of changes for patch 21.
===========================

- Removed "mapred.capacity-scheduler.queue.<queue-name>.max.map.slots" and "mapred.capacity-scheduler.queue.<queue-name>.max.reduce.slots"
entries.

- Removed testcases for the same.

> CapacityScheduler: It should be possible to set queue hard-limit beyond it's actual capacity
> --------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1105
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1105
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: MAPRED-1105-21-1.patch, MAPREDUCE-1105-version20.patch.txt
>
>
> Currently the CS caps a queue's capacity to it's actual capacity if a hard-limit is specified
to be greater than it's actual capacity. We should allow the queue to go upto the hard-limit
if specified.
> Also, I propose we change the hard-limit unit to be percentage rather than #slots.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message