Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1636146451.1226314124435.JavaMail.jira@brutus>
Date: Mon, 10 Nov 2008 02:48:44 -0800 (PST)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4523) Enhance how memory-intensive user
 tasks are handled
In-Reply-To: <1134360276.1225090124414.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4523:
------------------------------

    Attachment: HADOOP-4523-20081110.txt

bq. The latest patch kills only the last task that started if the sum total of all tasks' memory usage goes beyond the configured limit. Picking up only one task may or may not bring down the usage to within the configured limits.
Attaching a new patch to address this. TaskMemoryManagerThread now calls {{TaskTracker.findTaskToKill()}} repeatedly to find a few tasks with the least progress so as to bring down the total memory usage of all tasks falls below TT's limit, and then kills them. Modified the signature of {{TaskTracker.findTaskToKill()}} to {{TaskTracker.findTaskToKill(List<TaskAttempId> tasksToExclude)}} so as to help excluding tasks that are already marked for killing.

> Enhance how memory-intensive user tasks are handled
> ---------------------------------------------------
>
>                 Key: HADOOP-4523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4523
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Vinod K V
>         Attachments: HADOOP-4523-200811-05.txt, HADOOP-4523-200811-06.txt, HADOOP-4523-20081110.txt
>
>
> HADOOP-3581 monitors each Hadoop task to see if its memory usage (which includes usage of any tasks spawned by it and so on) is within a per-task limit. If the task's memory usage goes over its limit, the task is killed. This, by itself, is not enough to prevent badly behaving jobs from bringing down nodes. What is also needed is the ability to make sure that the sum total of VM usage of all Hadoop tasks does not exceed a certain limit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.