hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
Date Mon, 21 Sep 2020 09:59:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486849
]

ASF GitHub Bot logged work on HIVE-24179:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 21/Sep/20 09:58
            Start Date: 21/Sep/20 09:58
    Worklog Time Spent: 10m 
      Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r491922128



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##########
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
     String dbName = stripQuotes(root.getChild(0).getText());
     boolean isExtended = (root.getChildCount() > 1);
 
-    HiveTxnManager txnManager = null;
+    boolean useNewLocksFormat;
     try {
-      txnManager = TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+      HiveTxnManager txnManager = TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);

Review comment:
       Do we create txnManager instance here just to get the value of useNewLocksFormat flag?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 486849)
    Time Spent: 20m  (was: 10m)

> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
>                 Key: HIVE-24179
>                 URL: https://issues.apache.org/jira/browse/HIVE-24179
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>            Reporter: Stamatis Zampetakis
>            Assignee: Stamatis Zampetakis
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>         Attachments: summary.png
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement and monitoring
the heap memory of HS2. For a small heap (e.g., 2g) it only takes a few minutes before the
server crashes with OutOfMemory error such as the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at java.util.Arrays.copyOf(Arrays.java:3332)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
>         at java.lang.StringBuilder.append(StringBuilder.java:136)
>         at org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
>         at org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
>         at org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
>         at org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
>         at org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
>         at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
>         at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
>         at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
>         at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
>         at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
>         at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
>         at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
>         at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
>         at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
>         at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
>         at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
>         at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
>         at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
>         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
>         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
>         at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
>         at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
>         at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
>         at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
>         at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
>         at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
>         at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
>         at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by {{Hashtable$Entry}}
and {{ConcurrentHashMap$Node}} objects coming from Hive configurations referenced by {{DbTxnManager}}.

> The latter are not eligible for garbage collection since at [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
time they are passed implicitly in a callback  stored inside ShutdownHookManager.  
> When the {{DbTxnManager}} is closed properly the leak is not present since the callback
is [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
from ShutdownHookManager. 
> {{SHOW LOCKS}} statements create ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
[ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72])
a new {{TxnManager}} and they never close it leading to the memory leak.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message