hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J" <ha...@cloudera.com>
Subject Re: Review Request: MAPREDUCE-1347. Missing synchronization in MultipleOutputFormat.
Date Fri, 24 Jun 2011 20:00:36 GMT


> On 2011-06-24 19:39:07, Todd Lipcon wrote:
> > I think your issue with the test case is that you're calling tMOF.getRecordWriter()
inside the thread. So, each thread has its own record writer and hence has its own computing
map. You should call that from outside the thread, and just have the threads do the writing.

/me bangs head on the wall.

Thanks for catching it! Tests fail/pass now (w/o and w/ old style sync).

Posting patch to JIRA in a bit, unless there are further comments beyond that?


- Harsh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/953/#review910
-----------------------------------------------------------


On 2011-06-24 19:06:46, Harsh J wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/953/
> -----------------------------------------------------------
> 
> (Updated 2011-06-24 19:06:46)
> 
> 
> Review request for hadoop-mapreduce and Todd Lipcon.
> 
> 
> Summary
> -------
> 
> Used the makeComputingMap from Guava's MapMaker to provide a thread safe way of creating
a RecordWriter cache.
> 
> For some reason, the map is not really caching it and is instead trying to apply() over
and over again for the same key-value pairs.
> 
> 
> This addresses bug MAPREDUCE-1347.
>     http://issues.apache.org/jira/browse/MAPREDUCE-1347
> 
> 
> Diffs
> -----
> 
>   mapreduce/ivy.xml 85ee014 
>   mapreduce/ivy/libraries.properties 9d40aaa 
>   mapreduce/src/java/org/apache/hadoop/mapred/lib/MultipleOutputFormat.java b8944f1 
>   mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java
14c097d 
> 
> Diff: https://reviews.apache.org/r/953/diff
> 
> 
> Testing
> -------
> 
> Added a test case, but it fails with the current behavior of MapMaker's makeComputingMap()
(would pass if its alright)
> 
> 
> Thanks,
> 
> Harsh
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message