hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson" <sa...@pearsonwholesale.com>
Subject Re: Maps running after reducers complete successfully?
Date Fri, 03 Oct 2008 19:20:47 GMT
Do we not have an option to store the map results in hdfs?


"Owen O'Malley" <omalley@apache.org> wrote in 
message news:9E729175-3CBE-4B77-AC2C-8760FB492EF4@apache.org...
> It isn't optimal, but it is the expected behavior. In general when we 
> lose a TaskTracker, we want the map outputs regenerated so that any 
> reduces that need to re-run (including speculative execution). We  could 
> handle it as a special case if:
>   1. We didn't lose any running reduces.
>   2. All of the reduces (including speculative tasks) are done with 
> shuffling.
>   3. We don't plan on launching any more speculative reduces.
> If all 3 hold, we don't need to re-run the map tasks. Actually doing  so, 
> would be a pretty involved patch to the JobTracker/Schedulers.
> -- Owen

View raw message