hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kramer (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (MAPREDUCE-1924) Mappers running when reducers have finished
Date Fri, 13 Aug 2010 23:21:18 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Adam Kramer resolved MAPREDUCE-1924.

    Resolution: Duplicate

Yes, this is a dup of 1060.

> Mappers running when reducers have finished
> -------------------------------------------
>                 Key: MAPREDUCE-1924
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1924
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Adam Kramer
> Occasionally, I will run jobs for which some reducers are able to finish but there are
still mappers running. I understand why sometimes mappers restart themselves even after the
reduce phase has begun--too many fetch-failures, for example. But in today's case, ALL of
the reducers have succeeded and are done, so these mappers really ARE unnecessary...so it
is a bug that they are running.
> Then, I killed one of them to see what was up--it just restarted itself. So, it is another
bug that mappers don't know they're unnecessary when they're killed.
> My guess is that if one of these jobs, which clearly finished at least once, were to
die randomly a few times, it would take the whole job with it--even though the job has completed.
> Whenever all reduce tasks are complete, Hadoop should kill ALL remaining map tasks and
immediately move to finish the job.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message