hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <omal...@apache.org>
Subject Re: Is combiner and map in same JVM?
Date Tue, 14 Apr 2009 18:18:58 GMT

On Apr 14, 2009, at 11:10 AM, Saptarshi Guha wrote:

> Thanks. I am using 0.19, and to confirm, the map and combiner (in  
> the map jvm) are run in *different* threads at the same time?

And the change was actually made in 0.18. So since then, the combiner  
is called 0, 1, or many times on each key in both the mapper and the  
reducer. It is called in a separate thread from the base application  
in the map (in the reduce task, the combiner is only use during the  
shuffle).

> My native library is not thread safe, so I would have to implement  
> locks. Aaron's email gave me hope(since the map and combiner would  
> then be running sequentially), but this appears to make things  
> complicated.

Yes, you'll probably need locks around your code that isn't thread safe.

-- Owen



Mime
View raw message