From mapreduce-issues-return-13289-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org Tue May 04 21:37:36 2010 Return-Path: Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: (qmail 38732 invoked from network); 4 May 2010 21:37:36 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 4 May 2010 21:37:36 -0000 Received: (qmail 22595 invoked by uid 500); 4 May 2010 21:37:36 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 22566 invoked by uid 500); 4 May 2010 21:37:36 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 22558 invoked by uid 99); 4 May 2010 21:37:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 21:37:36 +0000 X-ASF-Spam-Status: No, hits=-1387.7 required=10.0 tests=ALL_TRUSTED,AWL X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 21:37:35 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o44LbFFL016851 for ; Tue, 4 May 2010 21:37:15 GMT Message-ID: <5395323.3011273009035314.JavaMail.jira@thor> Date: Tue, 4 May 2010 17:37:15 -0400 (EDT) From: "Chris Douglas (JIRA)" To: mapreduce-issues@hadoop.apache.org Subject: [jira] Commented: (MAPREDUCE-1276) Shuffle connection logic needs correction In-Reply-To: <948212691.1260351978213.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864019#action_12864019 ] Chris Douglas commented on MAPREDUCE-1276: ------------------------------------------ Oh, OK; I didn't know that. While this prevents the connection from being reused, it will still be garbage collected if it is simply closed, right? The current code prevents another Fetcher from fetching output from that host while the socket is "drained," which seems like an avoidable cost. Leaving the TCP connection open is regrettable... thoughts on this tradeoff? * If retained, would it be possible to update the comment on the drain code? The current explanation, "just in case", makes it sound like superstition instead of a deliberate choice * Since {{input.close()}} can also throw, would it make sense to do that within the try/catch or use {{IOUtils::cleanup}}? > Shuffle connection logic needs correction > ------------------------------------------ > > Key: MAPREDUCE-1276 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1276 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task > Affects Versions: 0.21.0 > Reporter: Jothi Padmanabhan > Assignee: Amareshwari Sriramadasu > Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-1276.txt > > > While looking at the code with Amareshwari, we realized that {{Fetcher#copyFromHost}} marks connection as successful when {{url.openConnection}} returns. This is wrong. Connection is done inside implicitly inside {{getInputStream}}; we need to split {{getInputStream}} into {{connect}} and {{getInputStream}} to handle the connection and read time out strategies correctly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.