hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" <prabakaran.1.natara...@nsn.com>
Subject HDFS Quota Error
Date Thu, 22 May 2014 07:04:29 GMT
Hi

When I run a query in Hive, I get below exception.  I noticed the error "No space left on
device".

Then I did "hadoop fs -count -q /var/local/hadoop" - which gave below output

none             inf            none             inf           69          275          288034318
hdfs://nnode:54310/var/local/hadoop

Why I am getting none and inf for space and remaining space quota?  Is this meaning is unlimited
space or is there is any space left?


I tried "hadoop dfsadmin -setSpaceQuota 100G /var/local/hadoop"  --  Not sure 100G is correct
or not?  How much I need to set and how to calculate this?

After setting 100G , I get the below output  for "hadoop fs -count -q /var/local/hadoop"

none             inf    107374182400    104408308039           73          286          297777777
hdfs://nnode:54310/var/local/hadoop


I have to wait to see whether 100G is going to give me an exception or not....


--------------


2014-05-22 10:48:43,585 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hadoop cause:java.io.FileNotFoundException: /var/local/hadoop/cache/mapred/local/taskTracker/hadoop/jobcache/job_201405211712_0625/jars/org/iq80/snappy/SnappyCompressor.class
(No space left on device)
2014-05-22 10:48:43,585 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201405211712_0625_r_000001_2:
java.io.FileNotFoundException: /var/local/hadoop/cache/mapred/local/taskTracker/hadoop/jobcache/job_201405211712_0625/jars/org/iq80/snappy/SnappyCompressor.class
(No space left on device)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:51)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:277)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
        at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
        at java.lang.Thread.run(Thread.java:744)


Thanks and Regards
Prabakaran.N  aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"





Mime
View raw message