spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: "java.io.IOException: No space left on device" while doing repartitioning in Spark
Date Tue, 05 May 2015 06:10:01 GMT
It could be filling up your /tmp directory. You need to set your
spark.local.dir or you can also specify SPARK_WORKER_DIR to another
location which has sufficient space.

Thanks
Best Regards

On Mon, May 4, 2015 at 7:27 PM, shahab <shahab.mokari@gmail.com> wrote:

> Hi,
>
> I am getting "No space left on device" exception when doing repartitioning
>  of approx. 285 MB of data  while these is still 2 GB space left ??
>
> does it mean that repartitioning needs more space (more than 2 GB) for
> repartitioning of 285 MB of data ??
>
> best,
> /Shahab
>
> java.io.IOException: No space left on device
> 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> 	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
> 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> 	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
> 	at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:473)
> 	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:569)
> 	at org.apache.spark.util.Utils$.copyStream(Utils.scala:331)
> 	at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$1.apply$mcVI$sp(ExternalSorter.scala:730)
> 	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> 	at org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:728)
> 	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:68)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:56)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
>
>

Mime
View raw message