hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1119) When tasks fail to report status, show tasks's stack dump before killing
Date Fri, 13 Nov 2009 02:30:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777341#action_12777341
] 

Aaron Kimball commented on MAPREDUCE-1119:
------------------------------------------

Vinod,

I'm not sure what markUnresponsiveTasks() has to do with anything; that's not called from
either fsError() or fatalError(), which are how the child signals an exception back to the
TT. 

Even still, I do not believe that calling dumpTaskStack() from within fsError/fatalError solves
the problem either. Signals are delivered asynchronously. Even if the signal were sent inside
of fsError() before its call to purgeTask(), there is an arbitrary amount of time before that
is handled by the child program. (Not to mention that purgeTask()'s logic in this patch is
'send SIGQUIT, wait, then SIGKILL' -- so doing the SIGQUIT before purgeTask() does not change
the effective serialization.) Furthermore, the act of printing the stack trace to stderr itself
takes an arbitrary amount of time. The child program after calling TaskUmbilicalProtocol.fsError()
or fatalError() will still immediately exit all threads; there is no guarantee that it will
service the signal before terminating naturally.

Based on the logic in org.apache.hadoop.mapred.Child, it is unlikely that the purgeTask()
associated with TUP.fsError()/fatalError() is actually killing the task; it's probably exiting
of its own accord first (otherwise, we'd be seeing the stack dumps since the SIGQUIT is serialized
before the SIGKILL as-is). So I don't think this removes any nondeterminism from the equation.
If we want child exceptions to *definitely* trigger a stack dump in the event of an exception,
then the child should enumerate its own threads, call Thread.getStackTrace() on them, and
print them to stderr itself to ensure that this process completes before exiting the JVM,
without relying on asynchronous signals.



> When tasks fail to report status, show tasks's stack dump before killing
> ------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1119
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1119
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron Kimball
>         Attachments: MAPREDUCE-1119.2.patch, MAPREDUCE-1119.3.patch, MAPREDUCE-1119.4.patch,
MAPREDUCE-1119.patch
>
>
> When the TT kills tasks that haven't reported status, it should somehow gather a stack
dump for the task. This could be done either by sending a SIGQUIT (so the dump ends up in
stdout) or perhaps something like JDI to gather the stack directly from Java. This may be
somewhat tricky since the child may be running as another user (so the SIGQUIT would have
to go through LinuxTaskController). This feature would make debugging these kinds of failures
much easier, especially if we could somehow get it into the TaskDiagnostic message

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message