Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <26051097.85091288118720915.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 14:45:20 -0400 (EDT)
From: "Joydeep Sen Sarma (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-2157) tasklauncher threads in
 TaskTracker can die because of unexpected interrupts
In-Reply-To: <29780790.83571288115540547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925083#action_12925083 ] 

Joydeep Sen Sarma commented on MAPREDUCE-2157:
----------------------------------------------

no i don't know the exact code path that caused this to happen. however:

- we had an instance of the TaskLauncher dying. the logs indicated it died because of an interrupted condition or interrupted exception. jobs got hung because of this.
- Thread.interrupted() state is a global state. any subsystem (like log4j) can set it. it's very unsafe to use this as a way of determining whether the thread should exit. we should gate thread exit on running flag.

note that all the other threads in the TaskTracker check the running flag to determine thread exit condition. I used the log4j code as a potential example of a subsystem that can set the interrupted flag - i am not even sure we are using asyncappender.

> tasklauncher threads in TaskTracker can die because of unexpected interrupts
> ----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2157
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2157
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>            Priority: Critical
>
> taskLauncher thread exits on interruptedException and on Interrupt conditions without checking for any shutdown flag:
>      while (!Thread.interrupted()) {
>         ...
>         } catch (InterruptedException e) { 
>           return; // ALL DONE                                                                                                                      
>         }
>      }
> If the interrupt happened because of reasons other than TaskTracker.close() - then the TaskTracker will look functional - but will not be able to schedule tasks anymore. worse - some tasks (that are in the launch queue) will hang indefinitely un UNASSIGNED state (the JobTracker will not even time them out). We have seen this cause jobs to hang indefinitely.
> It seems that the interrupted condition can be set by log4j (of which there are many calls inside TaskLauncher). See or instance: http://logging.apache.org/log4j/1.2/xref/org/apache/log4j/AsyncAppender.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.