hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1276) Shuffle connection logic needs correction
Date Wed, 05 May 2010 03:46:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864134#action_12864134
] 

Jothi Padmanabhan commented on MAPREDUCE-1276:
----------------------------------------------

While I added the "drain" code, we were debugging some issues with length corruption, which
have since been attributed to some bad jdk version (u13 IIRC). We have not seen any corruptions
since. So, I do not think the drain code is very critical, it is added to handle some really
rare situations. In the same vein, the drain code itself will be executed on such rare cases
as well. If input.close does the necessary job, I am fine with removing this code. And yes
+1 for using IOUtils::cleanup. 

> Shuffle connection logic needs correction 
> ------------------------------------------
>
>                 Key: MAPREDUCE-1276
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1276
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 0.21.0
>            Reporter: Jothi Padmanabhan
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: patch-1276.txt
>
>
> While looking at the code with Amareshwari, we realized that  {{Fetcher#copyFromHost}}
marks connection as successful when {{url.openConnection}} returns. This is wrong. Connection
is done inside implicitly inside {{getInputStream}}; we need to split {{getInputStream}} into
{{connect}} and {{getInputStream}} to handle the connection and read time out strategies correctly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message