hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2157) tasklauncher threads in TaskTracker can die because of unexpected interrupts
Date Tue, 26 Oct 2010 21:17:24 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925143#action_12925143
] 

Joydeep Sen Sarma commented on MAPREDUCE-2157:
----------------------------------------------

i grepped through both the tasktracker and jobtracker log and looked for anything matching
reinit. nothing there.

the TT log is very clear in indicating that the map launcher just stopped working at some
point (it never emerged from a wait call). The reduce task launcher kept working. unfortunately
i didn't take a stack dump of the TT before we restarted things. These are the log entries
after which the Launcher disappeared:

2010-10-23 22:16:59,381 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201010171815_32743_m_000019_0 task's s\
tate:UNASSIGNED
2010-10-23 22:16:59,381 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201010171815_32743_m_000019_0
which needs 1 slots
2010-10-23 22:16:59,381 INFO org.apache.hadoop.mapred.TaskTracker: TaskLauncher : Waiting
for 1 to launch attempt_201010171815_32743_m_000019_0, currently we have 0 free slots

looking back - the only way this could have happened is actually via the InterruptedException
code path (and not the interrupted() state). (since if we bailed out via the interrupted()
check - there would have been some more log entries filed after coming out of the wait. (But
that doesn't make sense as per Java's contract of InterruptedException).

> tasklauncher threads in TaskTracker can die because of unexpected interrupts
> ----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2157
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2157
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>            Priority: Critical
>
> taskLauncher thread exits on interruptedException and on Interrupt conditions without
checking for any shutdown flag:
>      while (!Thread.interrupted()) {
>         ...
>         } catch (InterruptedException e) { 
>           return; // ALL DONE                                                       
                                                              
>         }
>      }
> If the interrupt happened because of reasons other than TaskTracker.close() - then the
TaskTracker will look functional - but will not be able to schedule tasks anymore. worse -
some tasks (that are in the launch queue) will hang indefinitely un UNASSIGNED state (the
JobTracker will not even time them out). We have seen this cause jobs to hang indefinitely.
> It seems that the interrupted condition can be set by log4j (of which there are many
calls inside TaskLauncher). See or instance: http://logging.apache.org/log4j/1.2/xref/org/apache/log4j/AsyncAppender.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message