hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Gao <steve....@yahoo.com>
Subject [Help] Why "java.util.zip.ZipOutputStream" need to use /tmp?
Date Fri, 28 Aug 2009 18:19:05 GMT
would someone give us a hint? Thanks.
Why "java.util.zip.ZipOutputStream" need to use /tmp?

The hadoop version is 0.18.3 . Recently we got "out of space" issue. It's from "java.util.zip.ZipOutputStream".
We found that /tmp is full and after cleaning /tmp the problem is solved.

However why hadoop needs to use /tmp? We had already configured hadoop tmp to a local disk
in: hadoop-site.xml

<property>
  <name>hadoop.tmp.dir</name>
  <value> ... some large local disk ... </value>
</property>


Could it because java.util.zip.ZipOutputStream uses /tmp even if we configured hadoop.tmp.dir
to a large local disk?

The error log is here FYI:

java.io.IOException: No space left on device         
at java.io.FileOutputStream.write(Native Method)        
 at java.util.zip.ZipOutputStream.writeInt(ZipOutputStream.java:445)         
at java.util.zip.ZipOutputStream.writeEXT(ZipOutputStream.java:362)         
at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:220)         
at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:301)         
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:146)         
at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:321)         
at org.apache.hadoop.streaming.JarBuilder.merge(JarBuilder.java:79)         
at org.apache.hadoop.streaming.StreamJob.packageJobJar(StreamJob.java:628)         
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:843)         
at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:110)         
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:33)       
 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)       
 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)       
 
at java.lang.reflect.Method.invoke(Method.java:597)         
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)         
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)         
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)         
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)         
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)  
Executing Hadoop job failure




      


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message