hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Spiridonov (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-114) All reducer tasks are finished, while some mapper tasks are still running
Date Wed, 14 Oct 2009 02:01:31 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Alexey Spiridonov updated MAPREDUCE-114:

    Attachment: hadoop-bug-useless-task.png

I had encountered this very same issue.

The task in the attached screenshot had finished computing, and apparently had successfully
sent data to the reducers (I checked some reducer logs). However, it lost the tracker before
it notified the reducer about task completion. As a result, the job is stuck waiting for the
replacement map tasks to finish, even though they are not needed.

(I cut out hostnames from the screenshots for privacy reasons)

> All reducer tasks are finished, while some mapper tasks are still running
> -------------------------------------------------------------------------
>                 Key: MAPREDUCE-114
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-114
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Qi Liu
>         Attachments: hadoop-bug-overview.png, hadoop-bug-useless-task.png
> In a high load environment (i.e. multiple jobs are queued up to be executed), when all
reducer tasks of a job are finished, some mapper tasks of the same job may still running (possibly
re-executed due to lost task tracker, etc).
> This should not happen when a job has at least one reducer task. When all reducer tasks
are in SUCCEEDED state, the Hadoop JobTracker should kill all running mapper tasks, since
execution would be meaningless. The job should also switch to SUCCEEDED state when all reducer
tasks of that job succeeded successfully.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message