hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sivaramakrishnan Narayanan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
Date Mon, 02 Sep 2013 10:14:52 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755999#comment-13755999
] 

Sivaramakrishnan Narayanan commented on HIVE-3969:
--------------------------------------------------

[~navis] please see my blog on the subject [here|http://www.qubole.com/blog/qubole-hive-server/].
It is insufficient to reset the classloader as this doesn't close connections to jar files.
I've pasted the relevant part of the blog here:

There was one issue that was a little perplexing. After running for a week or so, QHS (Qubole
Hive Server) starting throwing “too many files open” exceptions. A quick lsof call confirmed
that there were numerous open file handles. Surprisingly, though, these all pointed to jar
files. After some investigation, we found that the URLClassLoader leaks file handles to jars
it opens (see this [link|http://management-platform.blogspot.in/2009/01/classloaders-keeping-jar-files-open.html]
for some dirty details). These are never garbage collected. We ended up using the non-standard
ClassLoaderUtil.releaseLoader to free up resources. Java 7 has a nicer solution for this where
URLClassLoader has a close method that performs the necessary cleanup.
                
> Session state for hive server should be cleanup
> -----------------------------------------------
>
>                 Key: HIVE-3969
>                 URL: https://issues.apache.org/jira/browse/HIVE-3969
>             Project: Hive
>          Issue Type: Bug
>          Components: Server Infrastructure
>            Reporter: Navis
>            Assignee: Navis
>            Priority: Trivial
>         Attachments: HIVE-3969.D8325.1.patch
>
>
> Currently "add jar" command by clients are adding child ClassLoader to worker thread
cumulatively, causing various problems.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message