hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1221) Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
Date Wed, 17 Mar 2010 19:25:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846541#action_12846541
] 

Scott Chen commented on MAPREDUCE-1221:
---------------------------------------

The patch has been changed over time. Here is a quick overall summary of what this patch does.

The purpose of this patch is to allow TaskTracker to kill/fail tasks based on the RSS memory
status.

We can set the following three different parameters
mapreduce.tasktracker.reserved.physicalmemory.mb
mapreduce.map.memory.physical.mb 
mapreduce.reduce.memory.physical.mb

They will determine the total allowed RSS memory for tasks and the limit for individual tasks.
If the total limit is violated, TaskTracker will kill the task with highest amount of memory
to relief the memory pressure.
If the per task limit is violated, TaskTracker will *fail* the task that violate the limit.

If the parameters are not set, there will not be any limit.

The implementation is mostly follow the virtual memory limiting logic that we already have.
And ProcfsBasedProcessTree also allow us to obtain physical memory of tasks.

The tests added are the following
TestTaskTrackerMemoryManager.testTasksCumulativelyExceedingTTPhysicalLimits()
TestTaskTrackerMemoryManager.testTasksBeyondPhysicalLimits()
They verifies the behavior of the cases when total memory limit and per task limit are triggered.
There is also a slight modification in TestTaskTrackerMemoryManager.testTasksWithinLimits()
to make sure the tasks within physical memory limit will run correctly.


> Kill tasks on a node if the free physical memory on that machine falls below a configured
threshold
> ---------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1221
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1221-v1.patch, MAPREDUCE-1221-v2.patch, MAPREDUCE-1221-v3.patch,
MAPREDUCE-1221-v4.patch
>
>
> The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds
a set of configured thresholds. I would like to extend this feature to enable killing tasks
if the physical memory used by that task exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using lots of memory,
the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs
from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits
(HADOOP-5883) were designed to address this problem. This works well when most map-reduce
jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory
for each task. On the other hand, if each task forks off mappers/reducers written in other
languages (python/php, etc), the total virtual memory usage of the process-subtree varies
greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message