spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-2255) create a self destructing iterator that releases records from hash maps
Date Tue, 24 Jun 2014 07:50:24 GMT

     [ https://issues.apache.org/jira/browse/SPARK-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reynold Xin updated SPARK-2255:
-------------------------------

    Summary: create a self destructing iterator that releases records from hash maps  (was:
create a self destructing iterator that release key value pairs from in-memory hash maps)

> create a self destructing iterator that releases records from hash maps
> -----------------------------------------------------------------------
>
>                 Key: SPARK-2255
>                 URL: https://issues.apache.org/jira/browse/SPARK-2255
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>
> This is a small thing to do that can help out with GC pressure. For aggregations (and
potentially joins), we don't really need to hold onto the key value pairs as soon as we have
iterate over them. We can create a self destructing iterator for AppendOnlyMap / ExternalAppendOnlyMap
that removes references to the key value pair as the iterator goes through records so those
memory can be freed quickly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message