hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4523) Enhance how memory-intensive user tasks are handled
Date Fri, 31 Oct 2008 06:08:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644220#action_12644220

Vivek Ratan commented on HADOOP-4523:

I'm proposing a couple of improvements: 
# The TT currently monitors each task (and its descendants) to see if that task's memory usage
goes over a per-task limit. The TT should additionally monitor to make sure that the sum of
memory used by all tasks should not go over a per-node limit (_tasks.maxmemory_). This situation
is unlikely t happen if schedulers consider memory judiciously when scheduling, but not all
schedulers may. In addition, the TT should pick tasks that ran last, when deciding what tasks
to kill. 
# I'm also proposing a way to specify _tasks.maxmemory_. This particular discussion is, however,
going on in HADOOP-4035, so we can continue discussion there. 

> Enhance how memory-intensive user tasks are handled
> ---------------------------------------------------
>                 Key: HADOOP-4523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4523
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Vinod K V
> HADOOP-3581 monitors each Hadoop task to see if its memory usage (which includes usage
of any tasks spawned by it and so on) is within a per-task limit. If the task's memory usage
goes over its limit, the task is killed. This, by itself, is not enough to prevent badly behaving
jobs from bringing down nodes. What is also needed is the ability to make sure that the sum
total of VM usage of all Hadoop tasks does not exceed a certain limit.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message