hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1221) Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
Date Thu, 19 Nov 2009 00:49:39 GMT
Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
---------------------------------------------------------------------------------------------------

                 Key: MAPREDUCE-1221
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: tasktracker
            Reporter: dhruba borthakur
            Assignee: dhruba borthakur


The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds a
set of configured thresholds. I would like to extend this feature to enable killing tasks
if the physical memory used by that task exceeds a certain threshold.

On a certain operating system (guess?), if user space processes start using lots of memory,
the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs
from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits
(HADOOP-5883) were designed to address this problem. This works well when most map-reduce
jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory
for each task. On the other hand, if each task forks off mappers/reducers written in other
languages (python/php, etc), the total virtual memory usage of the process-subtree varies
greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message