spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zuotingbing (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22793) Memory leak in Spark Thrift Server
Date Fri, 15 Dec 2017 09:37:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292248#comment-16292248
] 

zuotingbing commented on SPARK-22793:
-------------------------------------

ok, i will try to check it in master branch. Thanks.

> Memory leak in Spark Thrift Server
> ----------------------------------
>
>                 Key: SPARK-22793
>                 URL: https://issues.apache.org/jira/browse/SPARK-22793
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.2
>            Reporter: zuotingbing
>            Priority: Critical
>
> 1. Start HiveThriftServer2.
> 2. Connect to thriftserver through beeline.
> 3. Close the beeline.
> 4. repeat step2 and step 3 for several times, which caused the leak of Memory.
> we found there are many directories never be dropped under the path
> {code:java}
> hive.exec.local.scratchdir
> {code} and 
> {code:java}
> hive.exec.scratchdir
> {code} , as we know the scratchdir has been added to deleteOnExit when it be created.
So it means that the cache size of FileSystem deleteOnExit will keep increasing until JVM
terminated.
> In addition, we use 
> {code:java}
> jmap -histo:live [PID]
> {code} to printout the size of objects in HiveThriftServer2 Process, we can find the
object "org.apache.spark.sql.hive.client.HiveClientImpl" and "org.apache.hadoop.hive.ql.session.SessionState"
keep increasing even though we closed all the beeline connections, which caused the leak of
Memory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message