hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5883) TaskMemoryMonitorThread might shoot down tasks even if their processes momentarily exceed the requested memory
Date Mon, 25 May 2009 04:23:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712630#action_12712630
] 

Hemanth Yamijala commented on HADOOP-5883:
------------------------------------------

As per some offline discussions with Eric, Arun and Vinod, we decided to follow the approach
mentioned below for improving the monitoring logic:

The particular case in HADOOP-5059 indicates that any process that executes a program from
the Java code could occupy (momentarily) twice the amount of memory, due to the JVM's fork()+exec()
approach. This happens during the fork() and before the exec() completes.

- Hence, if a process tree is over twice the configured memory limit, then it is shot down
immediately.
- Also, if there are processes in the tree older than a certain interval and these processes
alone contribute to being over limit, then the task is shot down.

By corollary, this means that if there are no processes in the tree older than a certain interval
and the tree is still over limit, we give it a benefit of doubt to accomodate the case in
HADOOP-5059.

> TaskMemoryMonitorThread might shoot down tasks even if their processes momentarily exceed
the requested memory
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5883
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>
> Currently the TaskMemoryMonitorThread kills tasks as soon as it detects they are consuming
more memory than the max value specified. There are valid cases (see HADOOP-5059) where if
a program is executed from the task, it might momentarily occupy twice the amount of memory
for a short time. Ideally the monitoring thread should handle this case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message