spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ey-chih chow <eyc...@hotmail.com>
Subject RE: no space left at worker node
Date Sun, 08 Feb 2015 22:57:15 GMT
Hi Gen,
Thanks.  I save my logs in a file under /var/log.  This is the only place to save data.  Will
the problem go away if I use a better machine?
Best regards,
Ey-Chih Chow

Date: Sun, 8 Feb 2015 23:32:27 +0100
Subject: Re: no space left at worker node
From: gen.tang86@gmail.com
To: eychih@hotmail.com
CC: user@spark.apache.org

Hi,
I am sorry that I made a mistake. r3.large has only one SSD which has been mounted in /mnt.
Therefore this is no /dev/sdc.In fact, the problem is that there is no space in the under
/ directory. So you should check whether your application write data under this directory(for
instance, save file in file:///). 
If not, you can use watch du -sh to during the running time to figure out which directory
is expanding. Normally, only /mnt directory which is supported by SSD is expanding significantly,
because the data of hdfs is saved here. Then you can find the directory which caused no space
problem and find out the specific reason.
CheersGen


On Sun, Feb 8, 2015 at 10:45 PM, ey-chih chow <eychih@hotmail.com> wrote:



Thanks Gen.  How can I check if /dev/sdc is well mounted or not?  In general, the problem
shows up when I submit the second or third job.  The first job I submit most likely will succeed.
Ey-Chih Chow

Date: Sun, 8 Feb 2015 18:18:03 +0100
Subject: Re: no space left at worker node
From: gen.tang86@gmail.com
To: eychih@hotmail.com
CC: user@spark.apache.org

Hi,
In fact, /dev/sdb is /dev/xvdb. It seems that there is no problem about double mount. However,
there is no information about /mnt2. You should check whether /dev/sdc is well mounted or
not.The reply of Micheal is good solution about this type of problem. You can check his site.
CheersGen

On Sun, Feb 8, 2015 at 5:53 PM, ey-chih chow <eychih@hotmail.com> wrote:



Gen,
Thanks for your information.  The content of /etc/fstab at the worker node (r3.large) is:
#LABEL=/     /           ext4    defaults,noatime  1   1tmpfs       /dev/shm    tmpfs   defaults
       0   0devpts      /dev/pts    devpts  gid=5,mode=620  0   0sysfs       /sys        sysfs
  defaults        0   0proc        /proc       proc    defaults        0   0/dev/sdb     
  /mnt    auto    defaults,noatime,nodiratime,comment=cloudconfig 0       0/dev/sdc      
 /mnt2   auto    defaults,noatime,nodiratime,comment=cloudconfig 0       0
There is no entry of /dev/xvdb.
 Ey-Chih Chow
Date: Sun, 8 Feb 2015 12:09:37 +0100
Subject: Re: no space left at worker node
From: gen.tang86@gmail.com
To: eychih@hotmail.com
CC: user@spark.apache.org

Hi,
I fact, I met this problem before. it is a bug of AWS. Which type of machine do you use?
If I guess well, you can check the file /etc/fstab. There would be a double mount of /dev/xvdb.If
yes, you should1. stop hdfs2. umount /dev/xvdb at / 3. restart hdfs
Hope this could be helpful.CheersGen


On Sun, Feb 8, 2015 at 8:16 AM, ey-chih chow <eychih@hotmail.com> wrote:
Hi,



I submitted a spark job to an ec2 cluster, using spark-submit.  At a worker

node, there is an exception of 'no space left on device' as follows.



==========================================

15/02/08 01:53:38 ERROR logging.FileAppender: Error writing stream to file

/root/spark/work/app-20150208014557-0003/0/stdout

java.io.IOException: No space left on device

        at java.io.FileOutputStream.writeBytes(Native Method)

        at java.io.FileOutputStream.write(FileOutputStream.java:345)

        at

org.apache.spark.util.logging.FileAppender.appendToFile(FileAppender.scala:92)

        at

org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:72)

        at

org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)

        at

org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)

        at

org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)

        at

org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1311)

        at

org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)

===========================================



The command df showed the following information at the worker node:



Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/xvda1             8256920   8256456         0 100% /

tmpfs                  7752012         0   7752012   0% /dev/shm

/dev/xvdb             30963708   1729652  27661192   6% /mnt



Does anybody know how to fix this?  Thanks.





Ey-Chih Chow







--

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/no-space-left-at-worker-node-tp21545.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org




 		 	   		  

 		 	   		  

 		 	   		  
Mime
View raw message