hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1316) JobTracker holds stale references to retired jobs via speculated tips
Date Sat, 19 Dec 2009 09:21:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792829#action_12792829

Amar Kamat commented on MAPREDUCE-1316:

I think this scenario can be generalized further failed/killed jobs too. Consider the following
1) a job is running and the jobtracker schedules some tasks
2) even before the tasktracker with the newly scheduled task reports back, the job gets killed
or is failed because of errors
3) jobtracker goes ahead and launches cleanup task (if any) which also completes before the
task-tracker with scheduled tasks report back
4) the job completes and later retires
5) the tracker with newly launched task reports back 

The jobtracker simply ignores the updates as the job is retired. I think the problem is in
general with the tasks that are scheduled but the tasktacker reports only after the job retires.

> JobTracker holds stale references to retired jobs via speculated tips 
> ----------------------------------------------------------------------
>                 Key: MAPREDUCE-1316
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1316
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
> JobTracker fails to remove speculative tasks' mapping from _taskToTIPMap_ if the job
finishes and retires before the tracker (running the speculative tasks) reports back. In such
cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the
job is gone leading to memory leak.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message