hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
Date Tue, 29 Jul 2014 21:45:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078432#comment-14078432
] 

Allen Wittenauer commented on MAPREDUCE-1347:
---------------------------------------------

This appears to still be broken in 2.x Do we want to fix this?

> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
>                 Key: MAPREDUCE-1347
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.0.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J
>              Labels: concurrency
>         Attachments: MAPREDUCE-1347.r10.diff, MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff,
MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff,
MAPREDUCE-1347.r8.diff, MAPREDUCE-1347.r9.diff, mapreduce.1347.r1.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing
the recordWriters member. When using multithreaded mappers or reducers, this can result in
problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException.
Doing this more fine-grained than just synchronizing the whole method is probably a good idea,
so that multithreaded mappers can actually achieve parallelism writing into separate output
streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message