hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liyin Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2529) Recognize Jetty bug 1342 and handle it
Date Tue, 31 May 2011 02:16:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041381#comment-13041381
] 

Liyin Liang commented on MAPREDUCE-2529:
----------------------------------------

After upgrading to jetty 6.1.26, our product cluster met the same problem. Through observation,
we found TT will throw lots of "java.io.IOException: Broken pipe" when serve map-output and
Jetty print logs as follows in this case.

2011-05-30 00:11:06,389 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@6cf3a37f
Busy selector - injecting delay 3 times

So we just grep "Busy selector" from TT's log to detect this bug. 

> Recognize Jetty bug 1342 and handle it
> --------------------------------------
>
>                 Key: MAPREDUCE-2529
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2529
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.204.0, 0.23.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: jetty1342-20security.patch
>
>
> We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342).
The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot
of them throw exceptions and eventually fail. The only way we have found to get the TT out
of this state is to restart the TT.  This jira is to catch this particular exception (or perhaps
a configurable regex) and handle it in an automated way to either blacklist or shutdown the
TT after seeing it a configurable number of them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message