hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4035) Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory
Date Wed, 26 Nov 2008 13:08:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650991#action_12650991

Hemanth Yamijala commented on HADOOP-4035:

Comments on the scheduling portion:

- The config value for % of VMEM for RAM should be in capacity scheduler's conf - no ? I am
OK leaving it in JobConf as well. But in comments above, it seemed like we were going to define
this in the CapacityScheduler config.
- killJobsWithInvalidRequirements is looking at every single job in the waiting list of jobs
in a queue. This would be a very costly operation. I think instead, just before a scheduler
considers a job for scheduling, it should check for this, maybe by looking at the count of
running maps / reduces to decide if it's looking at it the first time.
- In getTaskFromJob, we are using a switch on Type being Map/Reduce. We should instead use
the pattern of defining an abstract method in TaskSchedulingMgr, and implementing it appropriately
for Map/Reduce in the specific type of TaskSchedulingMgr. So, pendingTasks would be an API
in TaskSchedulingMgr.
- If scheduling based on memory is disabled, we are logging at INFO level a statement regarding
this. This might get to be too verbose since most jobs aren't going to specify any memory
requirements. Switch to a debug level.
- Should we cache the value for the jobId -> value for running jobs. It will be low on
memory and will be more performant maybe if we use a HashMap. Maybe we should consider this
after benchmarking the capacity scheduler with and without this patch.
- We should definitely cache the value of the JobInProgress across computation of reservedPmem
and reservedVmem. Actually the lookup is done thrice - twice in getting reservedPmem.
- Rename usedUpVmem to getVmemReservedForTasks(taskTracker), likewise for usedUpPmem.
- isSchedulingBasedOnMemoryEnabled - for consistency, rename to isSchedulingBasedOnVmemEnabled
- It seems a little complicated to have the decision to lookup a job in the queue, or another
queue to be in TaskLookupResult. I think a simpler model would be for the TaskLookupResult
to only state the *reason* why it did not find a task, and then let callers decide what to
do next. This will make it simpler to change scheduling decisions if required later.
- For better readability, I am thinking if all the memory related scheduling should be moved
to a separate class that will be used by the CapacityTaskScheduler, maybe a class like MemoryMatcher.
This can have a package private method boolean matchesMemoryRequirements(JobInProgress jip,
TaskTrackerStatus ttStatus). Essentially move over everything starting from TTHasEnoughMemoryForJob
to this class. In future, if these algorithms seem reusable this class can move out of the
capacity scheduler into mapred itself.

> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements
and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-4035
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4035
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod K V
>            Priority: Blocker
>             Fix For: 0.20.0
>         Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt, HADOOP-4035-20081006.1.txt,
HADOOP-4035-20081006.txt, HADOOP-4035-20081008.txt, HADOOP-4035-20081121.txt, HADOOP-4035-20081126.1.txt
> HADOOP-3759 introduced configuration variables that can be used to specify memory requirements
for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler
in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on
a TT that uses more than the default amount of memory per slot can be viewed as effectively
using more than one slot, as it would decrease the amount of free memory on the TT by more
than the default amount while it runs. The scheduler should make the used capacity account
for this additional usage while enforcing limits, etc.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message