hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <omal...@apache.org>
Subject Re: Maps running after reducers complete successfully?
Date Thu, 02 Oct 2008 16:28:03 GMT
It isn't optimal, but it is the expected behavior. In general when we  
lose a TaskTracker, we want the map outputs regenerated so that any  
reduces that need to re-run (including speculative execution). We  
could handle it as a special case if:
   1. We didn't lose any running reduces.
   2. All of the reduces (including speculative tasks) are done with  
shuffling.
   3. We don't plan on launching any more speculative reduces.
If all 3 hold, we don't need to re-run the map tasks. Actually doing  
so, would be a pretty involved patch to the JobTracker/Schedulers.

-- Owen

Mime
View raw message