hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4035) Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory
Date Fri, 14 Nov 2008 10:42:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647577#action_12647577
] 

Vinod K V commented on HADOOP-4035:
-----------------------------------

Some details about configuration:
 - pmem denotes the limits of job and TT w.r.t physical memory
 - vmem denotes the limits of job and TT w.r.t virtual memory

h3. Job Configuration:

h4. Already addressed cases:
 - If pmem, and vmem are both specified, or both not specified(disabled/job doesn't care),
we use them as they are.
 - If pmem is not specified but vmem is specified, the above proposal is to adjust pmem to
be a percentage(say P) of vmem.

h4. Cases not addressed:
 - The proposal doesn't address the edge case when pmem is specified but vmem is not specified.
I propose that, in the similar vein as above, we adjust vmem to be (100/P) of pmem.

h3. TT Configuration:

h4. Already addressed cases:
 * If offsets for vmem and pmem are both specified,
      ** TT takes care of overflowing tasks itself by doing virtual memory management,
      ** Scheduler uses both vmem and pmem for scheduling and using the latter controls thrashing.
 * If offsets for both vmem and pmem are not specified,
      ** TT doesn't care about overflowing tasks and disables virtual memory management,
      ** Scheduler neglects scheduling based on vmem and pmem and cannot attempt avoiding
thrashing/task overflow.

h4. Cases not addressed:

 * If offset for vmem is specified but not for pmem. We have two alternative approaches here.
      ** We already calculate the total pmem reporting. So, take the pmem offset to be zero
and use the total pmem available and (vmem - vmemoffset) for scheduling. vmemoffset by default
will be zero. So, we just need another configuration and a field in TaskTrackerStatus to specify
whether vmem is disabled/enabled. Note that today we overload total vmem and by extension
vmemoffset to also specify enabling/disabling of task memory management.
      ** Only use vmem for scheduling and don't care about pmem and thus make no attempt to
avoid any possible thrashing.
* If offset for pmem is specified but not for vmem. We have two alternative approaches here
also.
      ** We already calculate the total vmem reporting. So, take the vmem offset to be zero
and use the total vmem available and (pmem - pmemoffset) for scheduling.
      ** Only use pmem for scheduling to avoid any possible thrashing, but don't care about
vmem and thus make no attempt to avoid any possible overflowing of tasks.

I prefer the first solutions to the later in both the above cases, as we are making our best
efforts to prevent thrashing and task overflowing in those solutions. Thoughts? 


> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements
and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4035
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4035
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod K V
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt, HADOOP-4035-20081006.1.txt,
HADOOP-4035-20081006.txt, HADOOP-4035-20081008.txt
>
>
> HADOOP-3759 introduced configuration variables that can be used to specify memory requirements
for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler
in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on
a TT that uses more than the default amount of memory per slot can be viewed as effectively
using more than one slot, as it would decrease the amount of free memory on the TT by more
than the default amount while it runs. The scheduler should make the used capacity account
for this additional usage while enforcing limits, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message