hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3581) Prevent memory intensive user tasks from taking down nodes
Date Tue, 02 Sep 2008 10:59:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627625#action_12627625

Hemanth Yamijala commented on HADOOP-3581:

Looks good.

Few minor comments:

- allProcessInfo is being used only within getProcessTree, hence it can be a local variable.
Otherwise, it will hold memory unnecessarily
- processTree should be cleared before every re-construction of the process tree.
- isAlive(pid): If there's an IOException, the code needs to return false.

- IMO, addTaskToMonitor is a better name than createProcessTreeInfo, as it better conveys
the meaning of the method. Likewise removeTaskToMonitor should replace removeProcessTreeInfo.

- I think the documentation need not include details about the classes etc, but should make
sense to the administrator. Something like: The interval, in milliseconds, the tasktracker
waits between two cycles of monitoring a task's memory usage.
- Also, generally the framework doesn't seem to define configuration variables with upper
case, so maybe mapred.tasktracker.taskmemorymanager.monitoring-interval ?
- Millisecs should be expanded to milliseconds.

> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>                 Key: HADOOP-3581
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3581
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: HADOOP-3581-final.txt, HADOOP-3581.20080901.2.txt, HADOOP-3581.6.0.txt,
patch_3581_0.1.txt, patch_3581_3.3.txt, patch_3581_4.3.txt, patch_3581_4.4.txt, patch_3581_5.0.txt,
> Sometimes user Map/Reduce applications can get extremely memory intensive, maybe due
to some inadvertent bugs in the user code, or the amount of data processed. When this happens,
the user tasks start to interfere with the proper execution of other processes on the node,
including other Hadoop daemons like the DataNode and TaskTracker. Thus, the node would become
unusable for any Hadoop tasks. There should be a way to prevent such tasks from bringing down
the node.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message