hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4035) Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory
Date Wed, 26 Nov 2008 07:08:46 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod K V updated HADOOP-4035:
------------------------------

    Attachment: HADOOP-4035-20081126.1.txt

Attaching a new patch.
 - Fixed the testcase TestCapacityScheduler and added tests to verify memory based scheduling.

 - Free memory computation is moved to the scheduler from the TT, so removed TestHighRAMJobs
and included TestTTMemoryReporting that verifies that memory values are reported correctly
by the TT.

 - While writing test-cases found and fixed a bug in JobQueuesManager.jobCompleted (+146).

    {code}
    if (qi.runningJobs.remove(oldInfo) != null) {
      qi.waitingJobs.remove(oldInfo);
    }
    {code}
Should be
   {code}
    if (qi.runningJobs.remove(oldInfo) == null) {
      qi.waitingJobs.remove(oldInfo);
    }
    {code}
This definitely needs to be fixed in previous versions too.

 - Added killJob(JobID) and getJob(JobID) methods to TaskTrackerManager. Scheduler needs access
to these methods. This is an interface change and so I had to modify TestJobQueueTaskScheduler
and TestFairScheduler too. Should we just change TaskTrackerManager to be an abstract class,
may be in another JIRA?

 - Minor documentation fixes and code-refactoring over the previous patch.

> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements
and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4035
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4035
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod K V
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt, HADOOP-4035-20081006.1.txt,
HADOOP-4035-20081006.txt, HADOOP-4035-20081008.txt, HADOOP-4035-20081121.txt, HADOOP-4035-20081126.1.txt
>
>
> HADOOP-3759 introduced configuration variables that can be used to specify memory requirements
for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler
in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on
a TT that uses more than the default amount of memory per slot can be viewed as effectively
using more than one slot, as it would decrease the amount of free memory on the TT by more
than the default amount while it runs. The scheduler should make the used capacity account
for this additional usage while enforcing limits, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message