spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@cs.berkeley.edu>
Subject Re: java.lang.OutOfMemoryError: Map failed
Date Fri, 20 Sep 2013 21:55:03 GMT
It looks like it is actually legitimately running out of memory. What does
application job do?



--
Reynold Xin, AMPLab, UC Berkeley
http://rxin.org



On Fri, Sep 20, 2013 at 11:57 AM, Reynold Xin <rxin@cs.berkeley.edu> wrote:

>
> Maybe you have a single key or a single partition that is extremely large?
>  Can you try log the gc information to see what else is going on?
>
>
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Thu, Sep 19, 2013 at 8:43 AM, Grega Kešpret <grega@celtra.com> wrote:
>
>> Hi!
>>
>> I have a simple job, basically sc.textFile(s3://...) -> filter -> map ->
>> reduceByKey() -> saveAsTextFile(s3://) that fails with OOM.
>>
>> Just before it fails I see a whole bunch of Full GC, each lasting 2-3
>> secs that don't recover any substantial heap space. Then I see
>>
>> Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server
>> VM warning: Attempt to deallocate stack guard pages failed.
>> Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack
>> guard pages failed.
>>
>> and finally
>>
>> 13/09/19 12:37:33 ERROR BlockManagerWorker: Exception handling buffer
>> message
>> java.io.IOException: Map failed
>>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>     at spark.storage.DiskStore.getBytes(DiskStore.scala:86)
>>     at spark.storage.BlockManager.getLocalBytes(BlockManager.scala:376)
>>     at
>> spark.storage.BlockManagerWorker.getBlock(BlockManagerWorker.scala:79)
>>     at
>> spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:58)
>>     at
>> spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:33)
>>     at
>> spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:33)
>>     at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>>     at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>>     at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>>     at
>> scala.collection.IndexedSeqLike$Elements.foreach(IndexedSeqLike.scala:54)
>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>>     at spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:12)
>>     at
>> scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
>>     at spark.storage.BlockMessageArray.map(BlockMessageArray.scala:12)
>>     at
>> spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:33)
>>     at
>> spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:23)
>>     at
>> spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:23)
>>     at
>> spark.network.ConnectionManager.spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:269)
>>     at
>> spark.network.ConnectionManager$$anon$5.run(ConnectionManager.scala:235)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>     at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>     at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>     ... 22 more
>>
>> I am not caching anything, and I am already using quite a large number of
>> numPartitions in reduceByKey (1000).
>>
>> The logs of one of the workers can be found here:
>>
>> https://dl.dropboxusercontent.com/u/921501/java.lang.OutOfMemoryError-Mapfailed.txt
>>
>> Grega
>> --
>> [image: Inline image 1]
>> *Grega Kešpret*
>> Analytics engineer
>>
>> Celtra — Rich Media Mobile Advertising
>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>
>
>

Mime
View raw message