hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-824) Support a hierarchy of queues in the capacity scheduler
Date Fri, 21 Aug 2009 06:42:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745827#action_12745827

Hemanth Yamijala commented on MAPREDUCE-824:

Looked at the test cases:
- Code seems duplicated between CapacitySchedulerUtils and CapacityTaskScheduler and TestContainerQueue.
- In some test cases, when we create a queue, it is already adding a child to the parent.
So, why do we need additional calls to addChildren ?
- What's the difference between testConfiguredCapacity and testMinCapacity ?
- The test cases testing scheduling are nice. The comments are out of sync a bit, and will
be hard to maintain. Instead I suggest that we assert what we are documenting in the tests
itself, so that they themselves read as comments, and will also always be in sync.
- As discussed, getCapacity() should not return max capacity any time. It should always return
the current capacity or limit, whichever is smaller. Otherwise, the sort order of queues would
be affected. 
- areTasksInQueueOverLimit should be changed to something along these lines:
      if (tsi.getMaxTaskLimit() > 0) {
        if (tsi.getNumSlotsOccupied() >= tsi.getCapacity()) {
          return true;
      if (tsi.getMaxCapacity() > 0) {
        if (tsi.getNumSlotsOccupied() >= tsi.getMaxCapacity()) {
          return true;
      return false;
- At the same time, testMaxCapacity should be removed. I would instead recommend a test case
that sets a max capacity on a queue, and checks scheduling honors the decision.

> Support a hierarchy of queues in the capacity scheduler
> -------------------------------------------------------
>                 Key: MAPREDUCE-824
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-824
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-824-1.patch, HADOOP-824-2.patch, HADOOP-824-3.patch
> Currently in Capacity Scheduler, cluster capacity is divided among the queues based on
the queue capacity. These queues typically represent an organization and the capacity of the
queue represents the capacity the organization is entitled to. Most organizations are large
and need to divide their capacity among sub-organizations they have. Or they may want to divide
the capacity based on a category or type of jobs they run. This JIRA covers the requirements
and other details to provide the above feature.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message