hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Reducers fail without messages on 20.205.0
Date Tue, 27 Dec 2011 07:19:21 GMT
I wouldn't use jvm reuse at this point. It's had a number of issues over time and I've consistently
switched it off for a long while now.


On Dec 26, 2011, at 2:50 PM, Markus Jelsma wrote:

> Hi,
>> Markus,
>> Good to know you fixed it now :)
>> Also consider raising reduce slowstart to a more suitable value, like
>> 80-85% -- this is pretty beneficial in both heap consumption and more
>> performant Reduce slots. The property name is
>> "mapred.reduce.slowstart.completed.maps" and carries a float value
>> (percentage).
> Is that going to help if one has a non-zero value for reusing JVM instances? 
> We have little concurrent jobs running because the process flow is linear and 
> we prefer the benefit of copying map output while it's running.
> For this particular job, however, it is not the case since all mappers finish 
> almost simultaneously so changing that value would not make a difference 
> right? 80% finishes within a matter of seconds.
> For the record, it's an Apache Nutch fetcher job configured to terminate after 
> 1 hour. This downloads URL's, hence the disabling of speculative execution.

View raw message