spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiangrui Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17822) JVMObjectTracker.objMap may leak JVM objects
Date Mon, 05 Dec 2016 17:59:59 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722886#comment-15722886
] 

Xiangrui Meng commented on SPARK-17822:
---------------------------------------

The issue comes with multiple RBackend connections. It is feasible to create multiple RBackend
sessions. But they share the same `JVMObjectTracker`. It cannot tell which JVM object is from
which RBackend. If an RBackend died without proper cleaning, we got a memory leak.

I will send a PR to make JVMObjectTracker a member variable of RBackend. There should be more
TODOs to allow concurrent RBackend sessions. But this would help solve the most critical issue.

> JVMObjectTracker.objMap may leak JVM objects
> --------------------------------------------
>
>                 Key: SPARK-17822
>                 URL: https://issues.apache.org/jira/browse/SPARK-17822
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>            Reporter: Yin Huai
>            Assignee: Xiangrui Meng
>         Attachments: screenshot-1.png
>
>
> JVMObjectTracker.objMap is used to track JVM objects for SparkR. However, we observed
that JVM objects that are not used anymore are still trapped in this map, which prevents those
object get GCed. 
> Seems it makes sense to use weak reference (like persistentRdds in SparkContext). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message