Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8CC149240 for ; Thu, 26 Jan 2012 10:56:03 +0000 (UTC) Received: (qmail 28449 invoked by uid 500); 26 Jan 2012 10:56:03 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 28363 invoked by uid 500); 26 Jan 2012 10:56:02 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 28350 invoked by uid 99); 26 Jan 2012 10:56:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 10:56:01 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 10:56:00 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 624631647AD for ; Thu, 26 Jan 2012 10:55:40 +0000 (UTC) Date: Thu, 26 Jan 2012 10:55:40 +0000 (UTC) From: "Harsh J (Commented) (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: <312206490.80674.1327575340404.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193753#comment-13193753 ] Harsh J commented on MAPREDUCE-1347: ------------------------------------ Bump, any more reviews on this one? > Missing synchronization in MultipleOutputFormat > ----------------------------------------------- > > Key: MAPREDUCE-1347 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 0.20.2, 0.21.0, 0.22.0 > Reporter: Todd Lipcon > Assignee: Harsh J > Fix For: 0.24.0 > > Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff, mapreduce.1347.r1.diff > > > MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams. > From what I can tell, the new API's MultipleOutputs seems not to have this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira