hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4801) ShuffleHandler can generate large logs due to prematurely closed channels
Date Fri, 16 Nov 2012 10:50:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498714#comment-13498714
] 

Hudson commented on MAPREDUCE-4801:
-----------------------------------

Integrated in Hadoop-Yarn-trunk #38 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/38/])
    MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels
(jlowe via bobby) (Revision 1410131)

     Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java

                
> ShuffleHandler can generate large logs due to prematurely closed channels
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4801
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4801
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.23.3, 2.0.1-alpha
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>             Fix For: 3.0.0, 2.0.3-alpha, 0.23.5
>
>         Attachments: MAPREDUCE-4801.patch
>
>
> We ran into an instance where many nodes on a cluster ran out of disk space because the
nodemanager logs were huge.  Examining the logs showed many, many shuffle errors due to either
ClosedChannelException or IOException from "Connection reset by peer" or "Broken pipe".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message