hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
Date Thu, 26 May 2011 17:55:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039830#comment-13039830
] 

Todd Lipcon commented on MAPREDUCE-1347:
----------------------------------------

- for the guava dependency, it should probably be conf="mapred->default" rather than releaseaudit
(otherwise the jars won't end up in the built package)
- the constant for version should be "guava.version" to match the naming convention
- this test case seems unlikely to actually fail even if the bug persisted -- if you revert
the fix and keep the test case, does it fail more than 50% of the time? Would it be possible
to directly instantiate the outputformat with some mocks without having to go through running
a whole job? Then you could hit it harder since you wouldn't have to touch disk.
- maybe add a comment explaining that the use of the computing map is to make sure this code
is thread-safe?

> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
>                 Key: MAPREDUCE-1347
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>         Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, mapreduce.1347.r1.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing
the recordWriters member. When using multithreaded mappers or reducers, this can result in
problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException.
Doing this more fine-grained than just synchronizing the whole method is probably a good idea,
so that multithreaded mappers can actually achieve parallelism writing into separate output
streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message