hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1316) JobTracker holds stale references to retired jobs via speculated tips
Date Fri, 18 Dec 2009 10:00:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792418#action_12792418

Amar Kamat commented on MAPREDUCE-1316:

Consider the following scenario 
1) a job is in its end stage waiting on few slow tasks to finish
2) the jobtracker goes ahead and speculates these tasks
3) even before the tasktracker with speculative task reports back, the non-speculative tasks
report as complete
4) jobtracker goes ahead and launches cleanup task (if any) which also completes before the
task-tracker with speculative tasks report back
5) the job completes and later retires
6) the tracker with speculative task reports back

In such a case if the tracker is not lost, then the taskToTIPMap will hold the mapping for
the speculative TIP forever which internally hold a reference to its job which has retired.

> JobTracker holds stale references to retired jobs via speculated tips 
> ----------------------------------------------------------------------
>                 Key: MAPREDUCE-1316
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1316
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Amar Kamat
> JobTracker fails to remove speculative tasks' mapping from _taskToTIPMap_ if the job
finishes and retires before the tracker (running the speculative tasks) reports back. In such
cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the
job is gone leading to memory leak.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message