spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory
Date Fri, 26 Feb 2016 12:00:22 GMT

    [ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168871#comment-15168871
] 

Sean Owen commented on SPARK-13510:
-----------------------------------

Without any detail about what you're  running, it's hard to help. It sounds like oyu're just
out of memory, which is not a bug. I don't see reason to believe there's an unreasonable amount
of memory being allocated

> Shuffle may throw FetchFailedException: Direct buffer memory
> ------------------------------------------------------------
>
>                 Key: SPARK-13510
>                 URL: https://issues.apache.org/jira/browse/SPARK-13510
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.0
>            Reporter: Hong Shen
>
> In our cluster, when I test spark-1.6.0 with a sql, it throw exception and failed.
> {code}
> org.apache.spark.shuffle.FetchFailedException: Direct buffer memory
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
> 	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:167)
> 	at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90)
> 	at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
> 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:759)
> 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:759)
> {code}
>   The reason is that when shuffle a big block(like 1G), task will allocate the same memory,
it will easily throw "FetchFailedException: Direct buffer memory".
>   If I add -Dio.netty.noUnsafe=true spark.executor.extraJavaOptions, it will throw 
> {code}
> java.lang.OutOfMemoryError: Java heap space
>         at io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:607)
>         at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237)
>         at io.netty.buffer.PoolArena.allocate(PoolArena.java:215)
>         at io.netty.buffer.PoolArena.allocate(PoolArena.java:132)
> {code}
>   
>   In mapreduce shuffle, it will firstly judge whether the block can cache in memery,
but spark doesn't. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message