hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Pang <p09...@gmail.com>
Subject Spill failed error
Date Tue, 21 Jul 2009 17:03:07 GMT
Hi users,

Please help with this one - I got an error at running a two - node cluster
on big files, the error is :

2365222 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher  -
Error message from task (map)
task_200907201602_0002_m_000010*java.io.IOException:
Spill failed*
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:425)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:96)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:81)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
    at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
*Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find any valid local directory for
taskTracker/jobcache/job_200907201602_0002/attempt_200907201602_0002_m_000010_0/output/spill0.out
*

*Cluster specification: *

Hadoop -version  0.18.3
Ubuntu -version  8.04

*etc/hosts*

127.0.0.1    localhost
127.0.1.1    wpang-laptop
192.168.0.1    master
192.168.0.2    slave

*hadoop-site.xml*

<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
</property>

<property>
  <name>mapred.job.tracker</name>
  <value>master:9001</value>
 </property>

<property>
  <name>dfs.replication</name>
  <value>2</value>
  </property>


*Hadoop web interface ( http://master:50070)
* Cluster Summary * * * 54 files and directories, 196 blocks = 250 total.
Heap Size is 20.5 MB / 963 MB (2%)
*  Capacity : 54.49 GB DFS Remaining : 18.21 GB DFS Used : 22.59 GB DFS
Used% : 41.45 % Live Nodes <http://localhost:50070/dfshealth.jsp#LiveNodes>
: 2 Dead Nodes <http://localhost:50070/dfshealth.jsp#DeadNodes> : 0
George

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message