hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
Date Tue, 09 Jul 2019 13:42:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated HIVE-21971:
------------------------------------
    Summary: HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary
functions + GenericUDF  (was: HS2 leaks classload due to `ReflectionUtils::CONSTRUCTOR_CACHE`
with temporary functions + GenericUDF)

> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions
+ GenericUDF
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-21971
>                 URL: https://issues.apache.org/jira/browse/HIVE-21971
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 2.3.4
>            Reporter: Rajesh Balamohan
>            Priority: Critical
>         Attachments: HIVE-21971.1.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from hadoop's
ReflectionUtils constructor cache issue (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use and this
causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not changed
much.
> Easiest way to repro would be to add a temp function which extends {{GenericUDF}}. In
{{FunctionRegistry::cloneGenericUDF,}} this would 
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in turn lands
up in COSNTRUCTOR_CACHE of ReflectionUtils. 
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR 'file:///home/test/udf/dummy.jar';
> select dummy();
> 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> 	at org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> 	at org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> 	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> 	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> 	at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> 	at org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was removed
in 2.x. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message