Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Tue, 31 May 2011 02:16:47 +0000 (UTC)
From: "Liyin Liang (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <1910157558.54866.1306808207714.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <994580154.39393.1306249607624.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-2529) Recognize Jetty bug 1342 and
 handle it
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041381#comment-13041381 ] 

Liyin Liang commented on MAPREDUCE-2529:
----------------------------------------

After upgrading to jetty 6.1.26, our product cluster met the same problem. Through observation, we found TT will throw lots of "java.io.IOException: Broken pipe" when serve map-output and Jetty print logs as follows in this case.

2011-05-30 00:11:06,389 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@6cf3a37f Busy selector - injecting delay 3 times

So we just grep "Busy selector" from TT's log to detect this bug. 

> Recognize Jetty bug 1342 and handle it
> --------------------------------------
>
>                 Key: MAPREDUCE-2529
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2529
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.204.0, 0.23.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: jetty1342-20security.patch
>
>
> We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot of them throw exceptions and eventually fail. The only way we have found to get the TT out of this state is to restart the TT.  This jira is to catch this particular exception (or perhaps a configurable regex) and handle it in an automated way to either blacklist or shutdown the TT after seeing it a configurable number of them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira