hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Palleti, Pallavi" <pallavi.pall...@corp.aol.com>
Subject RE: No Space Left On Device though space is available
Date Mon, 03 Aug 2009 05:13:00 GMT
No. These are production jobs which were working pretty fine and
suddenly, we started seeing these issues. And, if you see the error log,
the jobs are failing at the time of submission itself while copying the
application jar. And, when I see the client machine disk size and also
HDFS, it is only 60% full.

Thanks
Pallavi

-----Original Message-----
From: prashant ullegaddi [mailto:prashullegaddi@gmail.com] 
Sent: Monday, August 03, 2009 10:10 AM
To: common-user@hadoop.apache.org
Subject: Re: No Space Left On Device though space is available

Are you using any space on local nodes? When we indexed 1TB on 8 nodes,
we
were creating index on local file system and
then copying the same to DFS. It so happened that there wasn't any space
left. After that we started moving the index instead of copying it.
Everything worked fine.  Probably that could be a problem with your
application as well.

Thanks,
Prashant.

On Mon, Aug 3, 2009 at 10:05 AM, Pallavi Palleti <
pallavi.palleti@corp.aol.com> wrote:

> Hi all,
>
> We are having a 60 node cluster running hadoop-0.18.2. We are seeing
"No
> Space Left On Device" and the detailed error is
>  org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.RuntimeException: javax.xml.transfor
> m.TransformerException: java.io.IOException: No space left on device
>        at
> org.apache.hadoop.conf.Configuration.write(Configuration.java:996)
>        at
>
org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java
:530)
>        at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:196)
>        at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783)
>        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
>        at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>
>        at org.apache.hadoop.ipc.Client.call(Client.java:715)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>        at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
>        at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788)
>        at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
>
> Surprisingly, there is no space issue. Still, it is giving above
error. Can
> someone kindly let me know what could be the issue?
>
> Thanks
> Pallavi
>

Mime
View raw message