hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santiago PĂ©rez <elara...@gmail.com>
Subject Spill failed
Date Wed, 10 Feb 2010 08:53:40 GMT

Hej

I am running an aplication based on Nutch in a cluster with 1 master and 6
slaves in Amazon (with the same instances for all of them with 1.7GB RAM
memory)

My configuration is the following:

HADOOP_HEAPSIZE=1300
HADOOP_NAMENODE_OPTS=-Xmx400m
HADOOP_SECONDARYNAMENODE_OPTS=-Xmx400m
HADOOP_JOBTRACKER_OPTS=-Xmx400m
dfs.replication=3
mapred.map.tasks=6
mapred.reduce.tasks=6
mapred.child.java.opts=-Xmx950m

But in the second depth fetch, I got the following errors in some instances
(while the other ones seems they fetched correctly) :

2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - java.io.IOException: Spill
failed
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:822)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:907)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:670)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: Cannot run program "bash": java.io.IOException:
error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.run(Shell.java:134)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessImpl.start(ProcessImpl.java:65)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - ... 9 more
.
.
.
.
.
2010-02-10 03:18:31,463 WARN  mapred.TaskTracker - Error running child
java.io.IOException: Spill failed
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1085)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Cannot run program "bash":
java.io.IOException: error=12, Cannot allocate memory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
        at org.apache.hadoop.util.Shell.run(Shell.java:134)
        at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
        at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
        at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
        ... 9 more

Any idea??

Thanks in advance :) 
-- 
View this message in context: http://old.nabble.com/Spill-failed-tp27527092p27527092.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.


Mime
View raw message