From mapreduce-issues-return-22449-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 29 06:14:03 2011 Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C3A3D48A1 for ; Wed, 29 Jun 2011 06:14:03 +0000 (UTC) Received: (qmail 74388 invoked by uid 500); 29 Jun 2011 06:14:03 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 74048 invoked by uid 500); 29 Jun 2011 06:13:54 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 74006 invoked by uid 99); 29 Jun 2011 06:13:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 06:13:50 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 06:13:48 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id A6557438DFC for ; Wed, 29 Jun 2011 06:13:28 +0000 (UTC) Date: Wed, 29 Jun 2011 06:13:28 +0000 (UTC) From: "Harsh J (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: <448877844.1352.1309328008677.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Updated] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1347: ------------------------------- Attachment: MAPREDUCE-1347.r6.diff Comment's comment issues addressed :) The =C3=89 thing was due to Mac's compose key, I'd typed a "=E2=80=A6" ther= e. Removed. > Missing synchronization in MultipleOutputFormat > ----------------------------------------------- > > Key: MAPREDUCE-1347 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 0.20.2, 0.21.0, 0.22.0 > Reporter: Todd Lipcon > Assignee: Harsh J > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPR= EDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, mapredu= ce.1347.r1.diff > > > MultipleOutputFormat's RecordWriter implementation doesn't use synchroniz= ation when accessing the recordWriters member. When using multithreaded map= pers or reducers, this can result in problems where two threads will both t= ry to create the same file, causing AlreadyBeingCreatedException. Doing thi= s more fine-grained than just synchronizing the whole method is probably a = good idea, so that multithreaded mappers can actually achieve parallelism w= riting into separate output streams. > From what I can tell, the new API's MultipleOutputs seems not to have thi= s issue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira