Return-Path: Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: (qmail 60795 invoked from network); 10 Feb 2010 08:54:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 10 Feb 2010 08:54:14 -0000 Received: (qmail 95182 invoked by uid 500); 10 Feb 2010 08:54:13 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 95112 invoked by uid 500); 10 Feb 2010 08:54:12 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 95102 invoked by uid 500); 10 Feb 2010 08:54:11 -0000 Delivered-To: apmail-hadoop-core-dev@hadoop.apache.org Received: (qmail 95099 invoked by uid 99); 10 Feb 2010 08:54:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 08:54:11 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 08:54:01 +0000 Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1Nf8KK-00022L-Py for core-dev@hadoop.apache.org; Wed, 10 Feb 2010 00:53:40 -0800 Message-ID: <27527092.post@talk.nabble.com> Date: Wed, 10 Feb 2010 00:53:40 -0800 (PST) From: =?UTF-8?Q?Santiago_P=C3=A9rez?= To: core-dev@hadoop.apache.org Subject: Spill failed MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: elaragon@gmail.com X-Virus-Checked: Checked by ClamAV on apache.org Hej I am running an aplication based on Nutch in a cluster with 1 master and 6 slaves in Amazon (with the same instances for all of them with 1.7GB RAM memory) My configuration is the following: HADOOP_HEAPSIZE=1300 HADOOP_NAMENODE_OPTS=-Xmx400m HADOOP_SECONDARYNAMENODE_OPTS=-Xmx400m HADOOP_JOBTRACKER_OPTS=-Xmx400m dfs.replication=3 mapred.map.tasks=6 mapred.reduce.tasks=6 mapred.child.java.opts=-Xmx950m But in the second depth fetch, I got the following errors in some instances (while the other ones seems they fetched correctly) : 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - java.io.IOException: Spill failed 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:822) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:907) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:670) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.util.Shell.run(Shell.java:134) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at java.lang.UNIXProcess.(UNIXProcess.java:148) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at java.lang.ProcessImpl.start(ProcessImpl.java:65) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) 2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - ... 9 more . . . . . 2010-02-10 03:18:31,463 WARN mapred.TaskTracker - Error running child java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1085) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) at org.apache.hadoop.util.Shell.run(Shell.java:134) at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135) Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory at java.lang.UNIXProcess.(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 9 more Any idea?? Thanks in advance :) -- View this message in context: http://old.nabble.com/Spill-failed-tp27527092p27527092.html Sent from the Hadoop core-dev mailing list archive at Nabble.com.