hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3581) Prevent memory intensive user tasks from taking down nodes
Date Tue, 05 Aug 2008 09:04:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli updated HADOOP-3581:
--------------------------------------------

    Status: Open  (was: Patch Available)

Cancelling patch to incorporate Devaraj's review comments offline.
 - Remove all references to ProcfsBasedProcessTree in TT. TT should strictly use the abstract
interface ProcessTree. Only. So that implementing and using some other processTree doesn't
imply any changes to TT.
 - Let the pidFile be written during task creation *only if* memory management is enabled.
Despite this, hitting the disk to write pid to a file for each task might affect short jobs.
 - TT should get an instance of ProcessTree from a configuration variable - mapred.tasktracker.processtreeimpl
 - Make TT use a string for a pid instead of an integer.

> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>
>                 Key: HADOOP-3581
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3581
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: patch_3581_0.1.txt, patch_3581_3.3.txt, patch_3581_4.3.txt, patch_3581_4.4.txt,
patch_3581_5.0.txt
>
>
> Sometimes user Map/Reduce applications can get extremely memory intensive, maybe due
to some inadvertent bugs in the user code, or the amount of data processed. When this happens,
the user tasks start to interfere with the proper execution of other processes on the node,
including other Hadoop daemons like the DataNode and TaskTracker. Thus, the node would become
unusable for any Hadoop tasks. There should be a way to prevent such tasks from bringing down
the node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message