hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vaibhav Gumashta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-11408) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
Date Thu, 30 Jul 2015 04:52:04 GMT

     [ https://issues.apache.org/jira/browse/HIVE-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vaibhav Gumashta updated HIVE-11408:
------------------------------------
    Description: 
I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since it's
on top of a larger patch: HIVE-2573 that was added in 1.2). 

Basically, add jar creates a new classloader for loading the classes from the new jar and
adds the new classloader to the SessionState object of user's session, making the older one
its parent. Creating a temporary function uses the new classloader to load the class used
for the function. On closing a session, although there is code to close the classloader for
the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see
it holds on to the temporary function's class that should have gone away after the session
close. 

Steps to reproduce:
1.
{code}
jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
{code}


2. 
Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.


3. 
{code}
jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF';

{code}


4. 
Close the jdbc session.

5. 
Take the memory snapshot and verify that the new URLClassLoader is indeed there and is holding
onto the class it loaded (org.gumashta.udf.AUDF) for the session which we already closed.





  was:
I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since it's
on top of a larger patch: HIVE-2573 that was added in 1.2). 

Basically, add jar creates a new classloader for loading the classes from the new jar and
adds the new classloader to the SessionState object of user's session, making the older one
its parent. Creating a temporary function uses the new classloader to load the class used
for the function. On closing a session, although there is code to close the classloader for
the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see
it holds on to the temporary function's class that should have gone away after the session
close. 

Steps to reproduce:
1.
{code}
jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
{code}


2. 
Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.


3. 
{code}
jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF';

{code}


4. 
Take the memory snapshot and verify that the new URLClassLoader is indeed there and is holding
onto the class it loaded (org.gumashta.udf.AUDF).





> HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-11408
>                 URL: https://issues.apache.org/jira/browse/HIVE-11408
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.14.0
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>
> I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since
it's on top of a larger patch: HIVE-2573 that was added in 1.2). 
> Basically, add jar creates a new classloader for loading the classes from the new jar
and adds the new classloader to the SessionState object of user's session, making the older
one its parent. Creating a temporary function uses the new classloader to load the class used
for the function. On closing a session, although there is code to close the classloader for
the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see
it holds on to the temporary function's class that should have gone away after the session
close. 
> Steps to reproduce:
> 1.
> {code}
> jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
> {code}
> 2. 
> Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.
> 3. 
> {code}
> jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF';

> {code}
> 4. 
> Close the jdbc session.
> 5. 
> Take the memory snapshot and verify that the new URLClassLoader is indeed there and is
holding onto the class it loaded (org.gumashta.udf.AUDF) for the session which we already
closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message