hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kramer (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1924) Mappers running when reducers have finished
Date Wed, 07 Jul 2010 22:25:50 GMT
Mappers running when reducers have finished

                 Key: MAPREDUCE-1924
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1924
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Adam Kramer

Occasionally, I will run jobs for which some reducers are able to finish but there are still
mappers running. I understand why sometimes mappers restart themselves even after the reduce
phase has begun--too many fetch-failures, for example. But in today's case, ALL of the reducers
have succeeded and are done, so these mappers really ARE unnecessary...so it is a bug that
they are running.

Then, I killed one of them to see what was up--it just restarted itself. So, it is another
bug that mappers don't know they're unnecessary when they're killed.

My guess is that if one of these jobs, which clearly finished at least once, were to die randomly
a few times, it would take the whole job with it--even though the job has completed.

Whenever all reduce tasks are complete, Hadoop should kill ALL remaining map tasks and immediately
move to finish the job.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message