hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sreekanth Ramakrishnan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4178) The capacity scheduler's defaults for queues should be configurable.
Date Fri, 26 Sep 2008 05:25:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634755#action_12634755

Sreekanth Ramakrishnan commented on HADOOP-4178:

The check we are doing should exist, for one reason which I see, capacity for queue is configured
and user specifies it as negative. In that case user has specified a wrong capacity, user
has option of specifying only positive non-zero values. The -1 is passed by the system and
it is private and internal. I don't think it is a good idea for us to allow user to define
negative capacities. 

In my opinion the checks should always be present, but the question is what should be the
fail over mechanism, should we pass on default value or should we throw and exception and
stop the JT. I think we should do checks log.warn the negative values Log.Warn the errors
and continue processing as we now have defaults configured for these values.

With respect to computation calculation one place which I can see where rounding errors can
possibly happen is as follows : 3 queues with no GC configured. 

So values for GC would be 33, 33, 33 (this would mean 1% capacity not used).

Or case where 75% is to be allocated to 4 queues. You get 19 and 4 * 19 = 76 over allocation

With respect to testGCAllocationToQueues: the 25% is allocated to default you have remaining
75% to be allocated to the 4 queues. So 75/4 = 18.75. Eighteen is rounded up to 19. And not
all queue can have 19 as capacity, since we are internally balancing and computing the capacity
so that it does not exceed more than 100, One of the queues would get 18.

> The capacity scheduler's defaults for queues should be configurable.
> --------------------------------------------------------------------
>                 Key: HADOOP-4178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4178
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Owen O'Malley
>            Assignee: Sreekanth Ramakrishnan
>            Priority: Blocker
>             Fix For: 0.19.0
>         Attachments: HADOOP-4178-1.patch, HADOOP-4178-2.patch, HADOOP-4178-3.patch, HADOOP-4178-4.patch
> The default values for the queue attributes should be configurable rather than hard coded.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message