hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
Date Fri, 13 Dec 2013 16:48:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847640#comment-13847640
] 

Chris Nauroth commented on MAPREDUCE-5351:
------------------------------------------

No, AFAIK the JobInProgress_JobHistory.patch is not required, and this leak has been fixed
by the patches already committed to svn for MAPREDUCE-5351 and MAPREDUCE-5508.  If you're
seeing a hang, but not a memory leak, then this sounds like a different issue.  For example,
MAPREDUCE-5606 is another known bug that can cause a JT to hang.

> JobTracker memory leak caused by CleanupQueue reopening FileSystem
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5351
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 1.1.2
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>            Priority: Critical
>             Fix For: 1-win, 1.2.1
>
>         Attachments: JobInProgress_JobHistory.patch, MAPREDUCE-5351-1.patch, MAPREDUCE-5351-2.patch,
MAPREDUCE-5351-addendum-1.patch, MAPREDUCE-5351-addendum.patch, MAPREDUCE-5351.patch
>
>
> When a job is completed, closeAllForUGI is called to close all the cached FileSystems
in the FileSystem cache.  However, the CleanupQueue may run after this occurs and call FileSystem.get()
to delete the staging directory, adding a FileSystem to the cache that will never be closed.
> People on the user-list have reported this causing their JobTrackers to OOME every two
weeks.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message