hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1181) Enforce RSS memory limit in TaskMemoryManagerThread
Date Tue, 03 Nov 2009 18:57:32 GMT
Enforce RSS memory limit in TaskMemoryManagerThread

                 Key: MAPREDUCE-1181
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1181
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
          Components: tasktracker
    Affects Versions: 0.20.1
            Reporter: Scott Chen
             Fix For: 0.20.1

TaskMemoryManagerThread will periodically check the rss memory usage of every task. If the
memory usage exceeds the specified threshold, the task will be killed. Also if the total rss
memory of all tasks exceeds (total amount of memory - specified reserved memory). The task
with least progress will be killed to recover the reserved rss memory.

This is similar to the virtual memory limit provided by TaskMemoryManagerThread. But now the
limit is for rss memory. This new feature allow us to avoid page swapping which is prone to

The following are the related configurations
mapreduce.reduce.memory.rss.mb   // RSS memory allowed for a reduce task
mapreduce.map.memory.rss.mb       // RSS memory allowed for a map task
mapreduce.tasktracker.reserved.memory.rss.mb     // RSS memory reserved (not for tasks) on
a tasktracker

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message