Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Wed, 11 Jul 2012 18:01:37 +0000 (UTC)
From: "Rohini Palaniswamy (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <1094321587.36768.1342029697123.JavaMail.jiratomcat@issues-vm>
In-Reply-To: <1368534161.44457.1339007783219.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Commented] (HIVE-3098) Memory leak from large number of
 FileSystem instances in FileSystem.CACHE. (Must cache UGIs.)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411778#comment-13411778 ] 

Rohini Palaniswamy commented on HIVE-3098:
------------------------------------------

@Ashutosh
bq. to workaround underlying Filesystem issue, lets just disable the fs.cache via config parameter. Disabling cache will plug the memory leak by not filling FS cache.
  Disabling fs.cache is not going to matter. We are already getting a new FileSystem object for every request and the fs.cache is not used at all. The newly created FileSystem objects are still going to be in memory until garbage collected. And they don't get garbage collected for some reason (We have not analyzed what else is holding reference to them) as the experiments Mithun ran with fs.cache disabled did not make any difference. 

bq. Once FSContext apis are declared stable for other projects to consume, we can switch over to those where the promise is that underlying problem is fixed.
   As Daryn had mentioned in previous comments, this is not going to solve the problem at all.

We need to fix this. It is not good to have to restart the metastore server every week or two. I see two possible interim fixes until it is solved in core hadoop itself for all applications.
 1) Disable fs.cache and do fs.close() after every filesystem call in the code.  If there is a place to put the fs.close() after every request is executed it is easy. Or you might have to put fs.close() in too many places. Anyways it is not going to be good for performance.
  2) Add the fs.close() logic after a timeout to the current patch. Mithun was already working on this. I can't understand why you think it adds too much complexity. Other applications are already doing this. It will be difficult to answer to the Ops guys why we have a software that needs to be restarted often.
                
> Memory leak from large number of FileSystem instances in FileSystem.CACHE. (Must cache UGIs.)
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3098
>                 URL: https://issues.apache.org/jira/browse/HIVE-3098
>             Project: Hive
>          Issue Type: Bug
>          Components: Shims
>    Affects Versions: 0.9.0
>         Environment: Running with Hadoop 20.205.0.3+ / 1.0.x with security turned on.
>            Reporter: Mithun Radhakrishnan
>            Assignee: Mithun Radhakrishnan
>         Attachments: HIVE-3098.patch
>
>
> The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing the Oracle backend).
> The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads, in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had 1000000 instances of FileSystem, whose combined retained-mem consumed the entire heap.
> It boiled down to hadoop::UserGroupInformation::equals() being implemented such that the "Subject" member is compared for equality ("=="), and not equivalence (".equals()"). This causes equivalent UGI instances to compare as unequal, and causes a new FileSystem instance to be created and cached.
> The UGI.equals() is so implemented, incidentally, as a fix for yet another problem (HADOOP-6670); so it is unlikely that that implementation can be modified.
> The solution for this is to check for UGI equivalence in HCatalog (i.e. in the Hive metastore), using an cache for UGI instances in the shims.
> I have a patch to fix this. I'll upload it shortly. I just ran an overnight test to confirm that the memory-leak has been arrested.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira