hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4801) ShuffleHandler can generate large logs due to prematurely closed channels
Date Thu, 15 Nov 2012 23:17:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated MAPREDUCE-4801:
----------------------------------

    Attachment: MAPREDUCE-4801.patch

Patch to ignore errors from closed connections and IOExceptions related to the client closing
the connection early.
                
> ShuffleHandler can generate large logs due to prematurely closed channels
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4801
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4801
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.23.3, 2.0.1-alpha
>            Reporter: Jason Lowe
>            Priority: Critical
>         Attachments: MAPREDUCE-4801.patch
>
>
> We ran into an instance where many nodes on a cluster ran out of disk space because the
nodemanager logs were huge.  Examining the logs showed many, many shuffle errors due to either
ClosedChannelException or IOException from "Connection reset by peer" or "Broken pipe".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message