hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" <prabakaran.1.natara...@nsn.com>
Subject RE: HDFS Quota Error
Date Thu, 22 May 2014 09:05:58 GMT
Thanks for your reply.  We have more than 50% disk space.

Just FYI.. This is not a physical machine. Its vmware virtual machine.

Thanks and Regards
Prabakaran.N  aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"


From: ext Aitor Perez Cedres [mailto:aperez@pragsis.com]
Sent: Thursday, May 22, 2014 1:04 PM
To: user@hadoop.apache.org
Subject: Re: HDFS Quota Error


Maybe you are out of space in a local disk? That location[1] looks like the local dir where
MR places some intermediate files. Can you check the output of df -h on a shell?


[1] /var/local/hadoop/cache/mapred/local
On 22/05/14 09:04, Natarajan, Prabakaran 1. (NSN - IN/Bangalore) wrote:
Hi

When I run a query in Hive, I get below exception.  I noticed the error "No space left on
device".

Then I did "hadoop fs -count -q /var/local/hadoop" - which gave below output

none             inf            none             inf           69          275          288034318
hdfs://nnode:54310/var/local/hadoop

Why I am getting none and inf for space and remaining space quota?  Is this meaning is unlimited
space or is there is any space left?


I tried "hadoop dfsadmin -setSpaceQuota 100G /var/local/hadoop"  --  Not sure 100G is correct
or not?  How much I need to set and how to calculate this?

After setting 100G , I get the below output  for "hadoop fs -count -q /var/local/hadoop"

none             inf    107374182400    104408308039           73          286          297777777
hdfs://nnode:54310/var/local/hadoop


I have to wait to see whether 100G is going to give me an exception or not....


--------------


2014-05-22 10:48:43,585 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hadoop cause:java.io.FileNotFoundException: /var/local/hadoop/cache/mapred/local/taskTracker/hadoop/jobcache/job_201405211712_0625/jars/org/iq80/snappy/SnappyCompressor.class
(No space left on device)
2014-05-22 10:48:43,585 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201405211712_0625_r_000001_2:
java.io.FileNotFoundException: /var/local/hadoop/cache/mapred/local/taskTracker/hadoop/jobcache/job_201405211712_0625/jars/org/iq80/snappy/SnappyCompressor.class
(No space left on device)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:51)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
        at java.lang.Thread.run(Thread.java:744)


Thanks and Regards
Prabakaran.N  aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"





--
Aitor PĂ©rez
Big Data System Engineer

Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain

http://www.bidoop.es

Mime
View raw message