hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues maximum capacity calculation issues
Date Thu, 19 Jan 2012 19:18:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189306#comment-13189306

Thomas Graves commented on MAPREDUCE-3683:

However, if the parent's max-capacity is undefined i.e. UNLIMITED (-1), we have two choices:

 1. Use the parent's 'current' capacity to compute the max-capacity of the child at runtime
 2. Set the child's max-capacity to be UNLIMITED too.

I'm leaning towards (2) since it's simpler and easier to explain to users i.e. it's more consistent.
Otherwise, we run the risk of differing max-capacity (due to parent's capacity changing) which
seems unpredictable and hard to explain to users/admins.


I guess I expected if the parent wasn't set (its max-capacity defaults to -1), which I thought
implies a queue can use complete capacity of the cluster, so if a childs max-capacity was
set, then it would just use that childs max-capacity - which would be % of the entire cluster.
If you go with option 2 that means that if I set my max-capacity for my child queue it basically
does nothing, which isn't what I would expect either.

That isn't true. It just means there is no max-cap for the child and it can use as much capacity
as the parent queue gets, which might be capped by the parent/grandparent capacity/max-capacity
and so on. This seems like the correct behaviour. Right?

Ok, sorry I was wrong, it does enforce it, what threw me off here was the web UI shows the
absolute maximum capacity as 100% for the leaf queue and I expected this to be parents absolute
max capacity.  But I think that must be a web UI bug.  I will file separate issue for that.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-3683
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Thomas Graves
>            Assignee: Arun C Murthy
>            Priority: Blocker
>         Attachments: MAPREDUCE-3683.patch
> In the Capacity scheduler if you configure the queues to be hierarchical where you have
root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents
maximum capacity when calculate its own maximum capacity, instead it seems to use the parents
capacity.  Looking at the code its using the parents absoluteCapacity and I think it should
be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation
when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is
not configured, then it can use 100% of the cluster.  

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message