hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From z3r0c001 <ceo.co...@gmail.com>
Subject overeager reduce tasks
Date Tue, 24 Nov 2009 20:55:59 GMT
Hi,

this is a strange problem so please bear with me,  I submitted about 290
jobs on my queue on a 6 machine cluster (15 map slots, 15 reduce slots) ,
and everything was fine and dandy for first 15 hours, I noticed that some
jobs started reduce tasks even before single map had been executed, so out
of 15 reduce slots, 12 were taken up by one inactive job, with status as
 reduce -> copy, but no progress, so one active job had only 3 slots to run
with, and that slowed up overall execution,

please fell free to tell me if that was incoherent, but is there a way you
can tell queue to not execute reduce before map is done,  I've speculative
execution flag as FALSE.

thanks,

-dp

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message