hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhooti Verma <verma.vibho...@gmail.com>
Subject Re: Spill failed error
Date Thu, 23 Jul 2009 06:20:28 GMT
This mainly happens when you do not have enough space. Please clean up and
run again.

On Tue, Jul 21, 2009 at 10:33 PM, George Pang <p0941p@gmail.com> wrote:

> Hi users,
>
> Please help with this one - I got an error at running a two - node cluster
> on big files, the error is :
>
> 2365222 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher  -
> Error message from task (map)
> task_200907201602_0002_m_000010*java.io.IOException:
> Spill failed*
>    at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:425)
>    at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:96)
>    at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
>    at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
>    at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:81)
>    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
>    at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
> *Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not
> find any valid local directory for
>
> taskTracker/jobcache/job_200907201602_0002/attempt_200907201602_0002_m_000010_0/output/spill0.out
> *
>
> *Cluster specification: *
>
> Hadoop -version  0.18.3
> Ubuntu -version  8.04
>
> *etc/hosts*
>
> 127.0.0.1    localhost
> 127.0.1.1    wpang-laptop
> 192.168.0.1    master
> 192.168.0.2    slave
>
> *hadoop-site.xml*
>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
> </property>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://master:9000</value>
> </property>
>
> <property>
>  <name>mapred.job.tracker</name>
>  <value>master:9001</value>
>  </property>
>
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
>  </property>
>
>
> *Hadoop web interface ( http://master:50070)
> * Cluster Summary * * * 54 files and directories, 196 blocks = 250 total.
> Heap Size is 20.5 MB / 963 MB (2%)
> *  Capacity : 54.49 GB DFS Remaining : 18.21 GB DFS Used : 22.59 GB DFS
> Used% : 41.45 % Live Nodes <http://localhost:50070/dfshealth.jsp#LiveNodes
> >
> : 2 Dead Nodes <http://localhost:50070/dfshealth.jsp#DeadNodes> : 0
> George
>



-- 
cheers,
Vibhooti

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message