hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1060) JT should kill running maps when all the reducers have completed
Date Tue, 06 Oct 2009 04:43:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762500#action_12762500
] 

Jothi Padmanabhan commented on MAPREDUCE-1060:
----------------------------------------------

Here is one such scenario.
Towards the end of the reduce phase, speculative tasks were launched for some reducers. When
these speculative reducers tried to fetch map outputs, the TT  was unable to fetch the map
outputs, presumably because the disk had some issues by then. So, these maps were relaunched
in some other nodes. In the meanwhile, all the original reducers completed and the speculative
reducers killed. So, we have a situation where all the reducers were complete but some maps
are still running. 

> JT should kill running maps when all the reducers have completed
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1060
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1060
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Jothi Padmanabhan
>
> We have seen some situations where maps are still running when all the reducers have
completed. This could happen because of lost TT's, interplay of speculative tasks with bad
TT's etc. If the maps take a long time to run, it unnecessarily delays the job completion
time, as this map output is not required anyways. The JT should possibly kill running maps
when all the reducers have completed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message