hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: overeager reduce tasks
Date Tue, 24 Nov 2009 21:32:17 GMT
On 11/24/09 12:55 PM, "z3r0c001" <ceo.coder@gmail.com> wrote:
> this is a strange problem so please bear with me,  I submitted about 290
> jobs on my queue on a 6 machine cluster (15 map slots, 15 reduce slots) ,
> and everything was fine and dandy for first 15 hours, I noticed that some
> jobs started reduce tasks even before single map had been executed, so out
> of 15 reduce slots, 12 were taken up by one inactive job, with status as
>  reduce -> copy, but no progress, so one active job had only 3 slots to run
> with, and that slowed up overall execution,
> please fell free to tell me if that was incoherent, but is there a way you
> can tell queue to not execute reduce before map is done,  I've speculative
> execution flag as FALSE.

mapred.reduce.slowstart.completed.maps is the setting you are looking for.

See also https://issues.apache.org/jira/browse/MAPREDUCE-1184

View raw message