hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
Date Sat, 12 May 2012 11:19:48 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J updated MAPREDUCE-1347:
-------------------------------

    Attachment: MAPREDUCE-1347.r9.diff

Updated (rebased) patch that just fixes the issue and uses the right test location.

Removed ivy markers as those are irrelevant today (for guava is already available).

For moving the tests, since there seem to be lots of tests lying split like that, lets discuss/investigate
it over at MAPREDUCE-4253 instead.
                
> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
>                 Key: MAPREDUCE-1347
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J
>         Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff,
MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff,
MAPREDUCE-1347.r9.diff, mapreduce.1347.r1.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing
the recordWriters member. When using multithreaded mappers or reducers, this can result in
problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException.
Doing this more fine-grained than just synchronizing the whole method is probably a good idea,
so that multithreaded mappers can actually achieve parallelism writing into separate output
streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message