hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2721) Use job control for tasks (and therefore for pipes and streaming)
Date Thu, 18 Dec 2008 14:05:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Gummadi updated HADOOP-2721:
---------------------------------

    Attachment: HADOOP-2721-v2.patch

Attaching an intermediate patch HADOOP-2721-v2.patch that addresses:

(1) As we are moving pidfiles beyond the scope of memory management, we should move the pidfile
related api(getPidFilePath, removePidFile) out of TaskMemoryManager. 

Moved them to TaskTracker.

(2) destroySubprocessTree() should also ensure that the process-tree is indeed terminated/killed
by sending SIGKILL if needed. We should do something in the line of ProcfsBasedProcessTree.SigKillThread.


destroySubprocessTree() ensures that the process-tree is indeed terminated/killed by sending
SIGKILL. SigKillThread is moved from ProcfsBasedProcessTree to a separate file.

(3) I think the check to ensure that the process is the process group leader should be made
always. This adds one more level of confidence that we are indeed killing the right process.
But this should be done in destroySubprocessTree() and only when used by TT. In other cases,
this may not be the case. So we need to separate this may be by having separate api for destroySession
and destroySubprocessTree. TT would use destroySession. 

The check to ensure that the process is the process group leader is made always. Now we have
destroyProcessGroup() instead of destroySubProcessTree().


Testcase: Increased the memoryLimit to 30MB so that the subtree would be bigger(now it has
8 processes). Added a verification in TestProcfsBasedProcessTree that checks if all the sub-processes
are indeed wiped off. 

> Use job control for tasks (and therefore for pipes and streaming)
> -----------------------------------------------------------------
>
>                 Key: HADOOP-2721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2721
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/streaming, mapred, pipes
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>         Attachments: HADOOP-2721-v2.patch, HADOOP-2721.patch, HADOOP-2721.patch, HADOOP-2721.patch
>
>
> We should use the setsid command when the task is launched to create a new session. We
should be able to use the setsid program when we launch the bash process to create a new session.
That will allow us to kill the entire session with a single signal and remove the need for
the ping methods in both TaskTracker.Child and pipes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message