hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-16180) LLAP: Native memory leak in EncodedReader
Date Tue, 21 Mar 2017 05:49:41 GMT


Hive QA commented on HIVE-16180:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10480 tests executed
*Failed tests:*
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

This message is automatically generated.

ATTACHMENT ID: 12859687 - PreCommit-HIVE-Build

> LLAP: Native memory leak in EncodedReader
> -----------------------------------------
>                 Key: HIVE-16180
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: llap
>    Affects Versions: 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Sergey Shelukhin
>            Priority: Critical
>         Attachments:, FullGC-15GB-cleanup.png, Full-gc-native-mem-cleanup.png,
HIVE-16180.03.patch, HIVE-16180.04.patch, HIVE-16180.1.patch, HIVE-16180.2.patch, Native-mem-spike.png
> Observed this in internal test run. There is a native memory leak in Orc EncodedReaderImpl
that can cause YARN pmem monitor to kill the container running the daemon. Direct byte buffers
are null'ed out which is not guaranteed to be cleaned until next Full GC. To show this issue,
attaching a small test program that allocates 3x256MB direct byte buffers. First buffer is
null'ed out but still native memory is used. Second buffer user Cleaner to clean up native
allocation. Third buffer is also null'ed but this time invoking a System.gc() which cleans
up all native memory. Output from the test program is below
> {code}
> Allocating 3x256MB direct memory..
> Native memory used: 786432000
> Native memory used after data1=null: 786432000
> Native memory used after data2.clean(): 524288000
> Native memory used after data3=null: 524288000
> Native memory used without gc: 524288000
> Native memory used after gc: 0
> {code}
> Longer term improvements/solutions:
> 1) Use DirectBufferPool from hadoop or netty's
as direct byte buffer allocations are expensive (System.gc() + 100ms thread sleep).
> 2) Use HADOOP-12760 for proper cleaner invocation in JDK8 and JDK9

This message was sent by Atlassian JIRA

View raw message