hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11408) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
Date Fri, 04 Sep 2015 00:59:46 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730122#comment-14730122
] 

Thejas M Nair commented on HIVE-11408:
--------------------------------------

Change
{code}
// HIVE-11408, HADOOP-10513	    
// Hadoop's ReflectionUtils caches constructors for the classes it instantiated.	    
// In UDFs, this can result in classloaders not getting GCed for a temporary function,	  
 
// resulting in a PermGen leak when used extensively from HiveServer2
{code}

to 
{code}
// Hadoop's ReflectionUtils caches constructors for the classes it instantiated.	    
// In UDFs, this can result in classloaders not getting GCed for a temporary function,	  
 
// resulting in a PermGen leak when used extensively from HiveServer2. See profiler snapshot
in HADOOP-10513	    
{code}
?


> HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-11408
>                 URL: https://issues.apache.org/jira/browse/HIVE-11408
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.0.0, 1.1.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>         Attachments: HIVE-11408.1.patch
>
>
> I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since
it's on top of a larger patch: HIVE-2573 that was added in 1.2). 
> Basically, add jar creates a new classloader for loading the classes from the new jar
and adds the new classloader to the SessionState object of user's session, making the older
one its parent. Creating a temporary function uses the new classloader to load the class used
for the function. On closing a session, although there is code to close the classloader for
the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see
it holds on to the temporary function's class that should have gone away after the session
close. 
> Steps to reproduce:
> 1.
> {code}
> jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
> {code}
> 2. 
> Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.
> 3. 
> {code}
> jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF';

> {code}
> 4. 
> Close the jdbc session.
> 5. 
> Take the memory snapshot and verify that the new URLClassLoader is indeed there and is
holding onto the class it loaded (org.gumashta.udf.AUDF) for the session which we already
closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message