hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <>
Subject [jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
Date Sat, 29 Feb 2020 12:46:00 GMT


Hive QA commented on HIVE-21971:

Here are the results of testing the latest attachment:

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18095 tests executed
*Failed tests:*
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets] (batchId=1)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

This message is automatically generated.

ATTACHMENT ID: 12995015 - PreCommit-HIVE-Build

> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions
+ GenericUDF
> -------------------------------------------------------------------------------------------------------
>                 Key: HIVE-21971
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 2.3.4
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Critical
>         Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch, HIVE-21971.3.patch
> helped in moving away from hadoop's
ReflectionUtils constructor cache issue (
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use and this
causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not changed
> Easiest way to repro would be to add a temp function which extends {{GenericUDF}}. In
{{FunctionRegistry::cloneGenericUDF,}} this would 
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in turn lands
up in COSNTRUCTOR_CACHE of ReflectionUtils. 
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR 'file:///home/test/udf/dummy.jar';
> select dummy();
> 	at org.apache.hadoop.util.ReflectionUtils.newInstance(
> 	at org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(
> 	at org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(
> 	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(
> 	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(
> 	at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(
> 	at org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(
> 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was removed
in 2.x. 

This message was sent by Atlassian Jira

View raw message