hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1221) Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
Date Thu, 25 Feb 2010 08:08:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838237#action_12838237

dhruba borthakur commented on MAPREDUCE-1221:

> If tasks, and hence jobs, are not penalized i.e. killed rather than failed, what is the
incentive for the authors of the bad

>From my understanding, this JIRA is not about making users write good code. It is about
making nodes in the cluster not die and rot (via excessive swapping, aka MAPREDUCE-257) when
a huge-memory-consuming job comes along. It is also about letting other good jobs continue
to run peacefully as much as possible. In a true utility model, the service provider provides
service and it is upto the customer to write/deploy whenever code he wants to run on it (as
long as he pays for it, of course:-))

> So killing some tasks and letting others proceed may still be better than not killing.
That said, I agree that we can fine tune the policy

+1 for Hong's proposal. If somebody has a better policy of which tasks to kill when this situation
arises that will be great. The default policy is to kill tasks with the maximum memory used
and I think that is a very practical one, isn't it? 

> Kill tasks on a node if the free physical memory on that machine falls below a configured
> ---------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1221
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1221-v1.patch, MAPREDUCE-1221-v2.patch, MAPREDUCE-1221-v3.patch
> The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds
a set of configured thresholds. I would like to extend this feature to enable killing tasks
if the physical memory used by that task exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using lots of memory,
the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs
from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits
(HADOOP-5883) were designed to address this problem. This works well when most map-reduce
jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory
for each task. On the other hand, if each task forks off mappers/reducers written in other
languages (python/php, etc), the total virtual memory usage of the process-subtree varies
greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message