hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4523) Enhance how memory-intensive user tasks are handled
Date Thu, 06 Nov 2008 10:47:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vinod K V updated HADOOP-4523:

    Attachment: HADOOP-4523-200811-06.txt

Messed up the approach. Here's another patch that gets it right. It
 - monitors and kills any tasks that cross the individual tasks' limits they have.
 - kills the task with least progress(via tasktracker.findTaskToKill()) if even after the
first step the total memory usage across all tasks goes over the total usage allowed.
 - includes the tests testTasksWithNoIndividualLimitsButTotalUsageWithinTTLimits, testTasksWithinIndividualLimitsAndTotalUsageWithinTTLimits,
testTasksBeyondIndividualLimitsAndTotalUsageWithinTTLimits and testTasksWithinIndividualLimitsButTotalUsageBeyondTTLimits.

> Enhance how memory-intensive user tasks are handled
> ---------------------------------------------------
>                 Key: HADOOP-4523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4523
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Vinod K V
>         Attachments: HADOOP-4523-200811-05.txt, HADOOP-4523-200811-06.txt
> HADOOP-3581 monitors each Hadoop task to see if its memory usage (which includes usage
of any tasks spawned by it and so on) is within a per-task limit. If the task's memory usage
goes over its limit, the task is killed. This, by itself, is not enough to prevent badly behaving
jobs from bringing down nodes. What is also needed is the ability to make sure that the sum
total of VM usage of all Hadoop tasks does not exceed a certain limit.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message