Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <480442728.1261130418282.JavaMail.jira@brutus>
Date: Fri, 18 Dec 2009 10:00:18 +0000 (UTC)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-1316) JobTracker holds stale
 references to retired jobs via speculated tips
In-Reply-To: <254578906.1261130178427.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792418#action_12792418 ] 

Amar Kamat commented on MAPREDUCE-1316:
---------------------------------------

Consider the following scenario 
1) a job is in its end stage waiting on few slow tasks to finish
2) the jobtracker goes ahead and speculates these tasks
3) even before the tasktracker with speculative task reports back, the non-speculative tasks report as complete
4) jobtracker goes ahead and launches cleanup task (if any) which also completes before the task-tracker with speculative tasks report back
5) the job completes and later retires
6) the tracker with speculative task reports back

In such a case if the tracker is not lost, then the taskToTIPMap will hold the mapping for the speculative TIP forever which internally hold a reference to its job which has retired.

> JobTracker holds stale references to retired jobs via speculated tips 
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1316
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1316
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Amar Kamat
>
> JobTracker fails to remove speculative tasks' mapping from _taskToTIPMap_ if the job finishes and retires before the tracker (running the speculative tasks) reports back. In such cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the job is gone leading to memory leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.