spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-6076) Fix a potential OOM issue when StorageLevel is MEMORY_AND_DISK_SER
Date Fri, 01 May 2015 19:28:06 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-6076:
-----------------------------
    Assignee: Shixiong Zhu

> Fix a potential OOM issue when StorageLevel is MEMORY_AND_DISK_SER
> ------------------------------------------------------------------
>
>                 Key: SPARK-6076
>                 URL: https://issues.apache.org/jira/browse/SPARK-6076
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager
>    Affects Versions: 1.2.1
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>            Priority: Minor
>             Fix For: 1.4.0
>
>
> When StorageLevel is MEMORY_AND_DISK_SER, it will copy the content from file into memory,
then put it into MemoryStore. See: https://github.com/apache/spark/blob/dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L538
> However, if the file is bigger than the free memory, OOM will happen. A better approach
is testing if there is enough memory. If not, copyForMemory should not be created, since this
is an option operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message