Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C7414D27F for ; Wed, 5 Sep 2012 12:39:26 +0000 (UTC) Received: (qmail 82839 invoked by uid 500); 5 Sep 2012 12:39:21 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 82576 invoked by uid 500); 5 Sep 2012 12:39:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 82553 invoked by uid 99); 5 Sep 2012 12:39:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 12:39:20 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vasco.visser@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 12:39:12 +0000 Received: by obbtb18 with SMTP id tb18so31337obb.35 for ; Wed, 05 Sep 2012 05:38:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=yJfQYjHPY7w0uiYpAfS0vlStx0tuPnGx90q7y+nGhBY=; b=k4Y/shTlPbDthWsh81cRIIQo5eUU5fMKokpR7ClZO6iCNb2szSJYy/tyDNcmFZwtqg slC2pUZXUOAMknPSLqtAm3BkosiRQsQEwnWco/gJgCPxHeLeucjNYZNS8jSb6rw+XKCt Vf2FHHq0IjiwCtyUTIBhCss7DqcXqB+YcAXBS7Hl5dqrkR6D7+dTyjvZm1ZBfxpovA/L vLe/y4YNs4N4De346YjnmwjPG3fd3MUCManUoC8kOBgkwllX8vCk1ZMjkQQXizJJLvNt Lue0n1XwkVsV0sPg7mno6j/4EomEH+b3ZYNelYki9jsq+bco6bdsRX91tI3Fdg1RRfGy DStA== MIME-Version: 1.0 Received: by 10.182.75.100 with SMTP id b4mr16753894obw.12.1346848731583; Wed, 05 Sep 2012 05:38:51 -0700 (PDT) Received: by 10.60.11.72 with HTTP; Wed, 5 Sep 2012 05:38:51 -0700 (PDT) In-Reply-To: <50473F69.9030905@cs.uni-kassel.de> References: <503BAA39.9070607@gmail.com> <503E046D.1080904@gmail.com> <503E1EDC.80501@cs.uni-kassel.de> <503F400A.8060702@cs.uni-kassel.de> <5040A956.5090401@cs.uni-kassel.de> <50473DE9.8020008@cs.uni-kassel.de> <50473F69.9030905@cs.uni-kassel.de> Date: Wed, 5 Sep 2012 14:38:51 +0200 Message-ID: Subject: Re: mortbay, huge files and the ulimit From: Vasco Visser To: user@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just a guess, but could it simply be memory issue on reducer? In your adjusted program, maybe try running without -UseGCOverheadLimit and see if you still got OOM errors. >From the sun website: The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line. So basically if you get a GC overhead limit exceeded OOM error it means that your app is doing nothing but garbage collection, the vm is fighting against running out of memory. If that happens it might result in a timeout of the connection fetching the map data (i don't know if it can result in timeout, but I can imagine it could). Also, note that 63 GB on disk is probably going to be inflated in memory. So in general you cant say that 60 GB on disk will need 60GB of mem. Actually, some people us a rule of thumb to do x4 to get approx mem requirement. Just some ideas, not really a solution but maybe it helps you further. On Wed, Sep 5, 2012 at 2:02 PM, Bj=F6rn-Elmar Macek wrote: > Excuse me: in my last code section was some old code included. Here is it > again stripped of deprecated code: > > > package uni.kassel.macek.rtprep; > > > > import gnu.trove.iterator.TIntIterator; > import gnu.trove.map.hash.TIntObjectHashMap; > import gnu.trove.set.TIntSet; > > import java.io.IOException; > import java.util.ArrayList; > import java.util.Calendar; > import java.util.Iterator; > import java.util.List; > import java.util.StringTokenizer; > > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.fs.Path; > import org.apache.hadoop.io.IntWritable; > import org.apache.hadoop.io.LongWritable; > import org.apache.hadoop.io.Text; > import org.apache.hadoop.mapred.MapReduceBase; > import org.apache.hadoop.mapred.OutputCollector; > import org.apache.hadoop.mapred.Reporter; > import org.apache.hadoop.mapreduce.Job; > import org.apache.hadoop.mapreduce.Mapper; > import org.apache.hadoop.mapreduce.Reducer; > import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; > import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; > import org.apache.hadoop.util.GenericOptionsParser; > > public class RetweetApplication { > > public static class RetweetMapper1 extends Mapper Text> { > > public void map(Object key, Text value, > Context context) > throws IOException { > > String line =3D value.toString(); > > Tweet aTweet =3D new Tweet(line); > if(aTweet.getMention()!=3Dnull && aTweet.getMention().length(= )>0){ > /*aTweet.setRT(0); > output.collect(new Text(aTweet.toString()), new > Text(aTweet.toString()));*/ > aTweet.setRT(1); > try { > context.write(new Text(aTweet.getMention()), new > Text(aTweet.toString())); > } catch (InterruptedException e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > } > aTweet.setRT(0); > try { > context.write(new Text(aTweet.getAuthor()), new > Text(aTweet.toString())); > } catch (InterruptedException e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > > } > } > > public static class RetweetReducer1 extends Reducer > { > > public void reduce(Text atweet, Iterator tweets, > Context context) > throws IOException { > List originals =3D new ArrayList(); > List retweets =3D new ArrayList(); > while(tweets.hasNext()){ > Tweet aTweet =3D new Tweet(tweets.toString()); > originals.add(aTweet); > if(aTweet.getRT()>0) retweets.add(aTweet); > } > > for(Tweet originalTweet : originals){ > for(Tweet aRetweet : retweets){ > if(aRetweet.isRetweeted(originalTweet)) > { > aRetweet.setReactionTime(originalTweet.getDate()); > try { > context.write( > atweet, > new Text(aRetweet.toString())); > } catch (InterruptedException e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > } > } > } > } > } > > public static void main(String[] args) throws Exception { > Configuration conf =3D new Configuration(); > String[] otherArgs =3D new GenericOptionsParser(conf, > args).getRemainingArgs(); > if (otherArgs.length !=3D 2) { > System.err.println("Usage: wordcount "); > System.exit(2); > } > Job job =3D new Job(conf, "word count"); > > job.setJarByClass(RetweetApplication.class); > job.setMapperClass(RetweetMapper1.class); > job.setReducerClass(RetweetReducer1.class); > job.setOutputKeyClass(Text.class); > job.setOutputValueClass(Text.class); > FileInputFormat.addInputPath(job, new Path(otherArgs[0])); > FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); > System.exit(job.waitForCompletion(true) ? 0 : 1); > } > } > Am 05.09.2012 13:56, schrieb Bj=F6rn-Elmar Macek: > >> Hello again, >> >> i just wanted to keep you updated, in case anybody reads this and is >> interested in the mortbay-issue: >> I applied the wordcount example to a big input file and everything worke= d >> fine. I watched the TT logs all the time and no single mortbay-eof excep= tion >> occurred. The only possible explanation for me seemed to be, that my cod= e >> somehow produces the exceptions. So i started to keep my code as close t= o >> the wordcount example as possible: >> >> * first getting rid of any members within the reducer class, that may ta= ke >> lots of memory. >> * using the Context class instead of OutputCollector (in Mapper and >> Reducer) >> * removing any Comparator (and restructuring the code so it just >> calculates the same as with the sorting) >> * using standart HashPartitioner >> >> Especially the first 2 points made my job succeed in the end, ALTHOUGH t= he >> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node >> seems to complete all the work while the others are busy with their >> exceptions (at least from what i can see in the log). >> >> I REALLY dont know what i can do, to make this job run through like the >> wordcount example. I dont see any big difference between my program and = the >> example anymore. >> >> I added the code here: >> >> *Please* help me here. >> >> Best regards, >> Elmar >> >> >> I added the code here: >> ######################################## >> package uni.kassel.macek.rtprep; >> >> import java.util.Date; >> >> import org.apache.hadoop.conf.Configuration; >> import org.apache.hadoop.conf.Configured; >> import org.apache.hadoop.fs.Path; >> import org.apache.hadoop.io.Text; >> import org.apache.hadoop.mapred.FileInputFormat; >> import org.apache.hadoop.mapred.FileOutputFormat; >> import org.apache.hadoop.mapred.JobClient; >> import org.apache.hadoop.mapred.JobConf; >> import org.apache.hadoop.mapred.TextOutputFormat; >> import org.apache.hadoop.util.Tool; >> import org.apache.hadoop.util.ToolRunner; >> >> public class RetweetApplication extends Configured implements Tool{ >> >> public static int DEBUG =3D -1; >> private Configuration jobConf; >> >> //static String INPUT =3D "/home/fb16/bmacek/HADOOP-INPUT"; >> //static String OUTPUT =3D "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new >> Date()).toString(); >> >> public static void main(String[] args) { >> int res=3D-1; >> try { >> res =3D ToolRunner.run(new Configuration(), new >> RetweetApplication(), args); >> } catch (Exception e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> System.exit(res); >> >> } >> >> public void setConf(Configuration conf) { >> jobConf =3D conf; >> >> } >> >> public Configuration getConf() { >> return jobConf; >> } >> >> public int run(String[] args) throws Exception { >> JobClient client =3D new JobClient(); >> JobConf conf =3D new JobConf(RetweetApplication.class); >> >> >> if(DEBUG > 0){ >> conf.set("mapred.job.tracker", "local"); >> conf.set("fs.default.name", "file:///"); >> conf.set("dfs.replication", "1"); >> } >> >> //conf.set("mapred.child.java.opts","-Xmx3000m >> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit"); >> conf.set("mapred.child.ulimit","100000000"); >> >> FileInputFormat.setInputPaths(conf, new Path(args[0])); >> FileOutputFormat.setOutputPath(conf, new Path(args[1])); >> >> >> //conf.setOutputKeyClass(Text.class); >> //conf.setOutputValueClass(Text.class); >> conf.setMapOutputKeyClass(Text.class); >> conf.setMapOutputValueClass(Text.class); >> >> conf.setMapperClass(RetweetMapper.class); >> //conf.setPartitionerClass(TweetPartitioner.class); >> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class)= ; >> >> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.cla= ss); >> conf.setReducerClass(RetweetReducer.class); >> >> conf.setOutputFormat(TextOutputFormat.class); >> >> client.setConf(conf); >> try { >> JobClient.runJob(conf); >> } catch (Exception e) { >> e.printStackTrace(); >> } >> return 0; >> } >> } >> */ >> >> package uni.kassel.macek.rtprep; >> >> >> >> import gnu.trove.iterator.TIntIterator; >> import gnu.trove.map.hash.TIntObjectHashMap; >> import gnu.trove.set.TIntSet; >> >> import java.io.IOException; >> import java.util.ArrayList; >> import java.util.Calendar; >> import java.util.Iterator; >> import java.util.List; >> import java.util.StringTokenizer; >> >> import org.apache.hadoop.conf.Configuration; >> import org.apache.hadoop.fs.Path; >> import org.apache.hadoop.io.IntWritable; >> import org.apache.hadoop.io.LongWritable; >> import org.apache.hadoop.io.Text; >> import org.apache.hadoop.mapred.MapReduceBase; >> import org.apache.hadoop.mapred.OutputCollector; >> import org.apache.hadoop.mapred.Reporter; >> import org.apache.hadoop.mapreduce.Job; >> import org.apache.hadoop.mapreduce.Mapper; >> import org.apache.hadoop.mapreduce.Reducer; >> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; >> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; >> import org.apache.hadoop.util.GenericOptionsParser; >> >> public class RetweetApplication { >> >> public static class RetweetMapper1 extends Mapper> Text> { >> >> public void map(Object key, Text value, >> Context context) >> throws IOException { >> >> String line =3D value.toString(); >> >> Tweet aTweet =3D new Tweet(line); >> if(aTweet.getMention()!=3Dnull && >> aTweet.getMention().length()>0){ >> /*aTweet.setRT(0); >> output.collect(new Text(aTweet.toString()), new >> Text(aTweet.toString()));*/ >> aTweet.setRT(1); >> try { >> context.write(new Text(aTweet.getMention()), new >> Text(aTweet.toString())); >> } catch (InterruptedException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> } >> aTweet.setRT(0); >> try { >> context.write(new Text(aTweet.getAuthor()), new >> Text(aTweet.toString())); >> } catch (InterruptedException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> >> } >> } >> >> public static class RetweetReducer1 extends Reducer> Text> { >> >> public void reduce(Text atweet, Iterator tweets, >> Context context) >> throws IOException { >> List originals =3D new ArrayList(); >> List retweets =3D new ArrayList(); >> while(tweets.hasNext()){ >> Tweet aTweet =3D new Tweet(tweets.toString()); >> originals.add(aTweet); >> if(aTweet.getRT()>0) retweets.add(aTweet); >> } >> >> for(Tweet originalTweet : originals){ >> for(Tweet aRetweet : retweets){ >> if(aRetweet.isRetweeted(originalTweet)) >> { >> aRetweet.setReactionTime(originalTweet.getDate()); >> try { >> context.write( >> atweet, >> new Text(aRetweet.toString())); >> } catch (InterruptedException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> } >> } >> } >> } >> } >> >> public static void main(String[] args) throws Exception { >> Configuration conf =3D new Configuration(); >> String[] otherArgs =3D new GenericOptionsParser(conf, >> args).getRemainingArgs(); >> if (otherArgs.length !=3D 2) { >> System.err.println("Usage: wordcount "); >> System.exit(2); >> } >> Job job =3D new Job(conf, "tweets"); >> job.setJarByClass(RetweetApplication.class); >> job.setMapperClass(RetweetMapper1.class); >> job.setReducerClass(RetweetReducer1.class); >> job.setOutputKeyClass(Text.class); >> job.setOutputValueClass(Text.class); >> FileInputFormat.addInputPath(job, new Path(otherArgs[0])); >> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); >> System.exit(job.waitForCompletion(true) ? 0 : 1); >> } >> } >> Am 31.08.2012 14:08, schrieb Bj=F6rn-Elmar Macek: >>> >>> Hi all, >>> >>> i got some more information: >>> >>> It is always only one TT, where mortbay errors do not occur: all other >>> slaves fail relatively fast after the reduce process started: maybe it = is a >>> comparator issue (since the operation is MAPRED_SHUFFLE) >>> And it has nothing to do with the local configuration of the server, >>> since >>> 1) if i rerun the job on the same set of servers, the ONE working diffe= rs >>> from one job to another. >>> 2) the same happens even when i am using a completely different set of >>> servers. >>> >>> Who is capable to solve this unsolved mystery? :D >>> >>> Am 30.08.2012 12:27, schrieb Bj=F6rn-Elmar Macek: >>>> >>>> Hi, >>>> >>>> is there anybody, who knows more about this issue: it has already been >>>> recently marked here: >>>> https://issues.apache.org/jira/browse/MAPREDUCE-5 >>>> >>>> I really want to do something about it, if i knew how... i tried so ma= ny >>>> different setup parameters and JVM options and nothing did the trick..= . >>>> >>>> It was opened 2009 and gets still updated (2012): I mean - this does N= OT >>>> look OK, does it? The whole TT seems like a big time waster: >>>> >>>> 2012-08-30 12:05:22,582 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, cliI= D: >>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852 >>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: >>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - inj= ecting >>>> delay2 times >>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: >>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>> recreating selector 2 times, canceled keys 70 times >>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: >>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 >>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:08:39,236 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>> attempt_201208301039_0001_m_000204_0, duration: 3209410 >>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput >>>> java.lang.IllegalStateException: Committed >>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3945) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> 2012-08-30 12:10:27,109 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, cliI= D: >>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517 >>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: >>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 >>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:14:29,740 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>> attempt_201208301039_0001_m_000236_0, duration: 3746212 >>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput >>>> java.lang.IllegalStateException: Committed >>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3945) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> 2012-08-30 12:16:28,503 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, cliI= D: >>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393 >>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: >>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - inj= ecting >>>> delay1 times >>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: >>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>> recreating selector 1 times, canceled keys 69 times >>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: >>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 >>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed : >>>> org.mortbay.jetty.EofException >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGene= rator.java:551) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.jav= a:572) >>>> at >>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012= ) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:651) >>>> at >>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.jav= a:580) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3916) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> Caused by: java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>> at >>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.j= ava:221) >>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>> ... 27 more >>>> >>>> 2012-08-30 12:20:01,205 INFO >>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 141.51.205.130:= 35497, >>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>> attempt_201208301039_0001_m_000241_0, duration: 5405031 >>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput >>>> java.lang.IllegalStateException: Committed >>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>> at >>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracke= r.java:3945) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> at >>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1221) >>>> at >>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServ= er.java:835) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH= andler.java:1212) >>>> at >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:39= 9) >>>> at >>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java= :216) >>>> at >>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:18= 2) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:76= 6) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>> at >>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandl= erCollection.java:230) >>>> at >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:15= 2) >>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542= ) >>>> at >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon= nection.java:928) >>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212= ) >>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404= ) >>>> at >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.jav= a:410) >>>> at >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja= va:582) >>>> >>>> Am 29.08.12 15:53, schrieb Bj=F6rn-Elmar Macek: >>>>> >>>>> Hi there, >>>>> >>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv >>>>> file on 20 physically distinct nodes with 15GB each: >>>>> >>>>> While the mapping works just fine and is low cost, the reducer does t= he >>>>> main work: holding a hashmap with elements to join with and finding j= oin >>>>> tuples for evry incoming key-value-pair. >>>>> >>>>> The jobs works perfectly on small files with 2 gigabytes, but starts = to >>>>> get "unstable" as the file size goes up: this becomes evident with a = look >>>>> into the tasktracker's logs saying: >>>>> >>>>> ERROR org.mortbay.log: /mapOutput >>>>> java.lang.IllegalStateException: Committed >>>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>>> at >>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack= er.java:3945) >>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>> at >>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511= ) >>>>> at >>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet= Handler.java:1221) >>>>> at >>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpSer= ver.java:835) >>>>> at >>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet= Handler.java:1212) >>>>> at >>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:3= 99) >>>>> at >>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.jav= a:216) >>>>> at >>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:1= 82) >>>>> at >>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:7= 66) >>>>> at >>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>> at >>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHand= lerCollection.java:230) >>>>> at >>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:1= 52) >>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>> at >>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:54= 2) >>>>> at >>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCo= nnection.java:928) >>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:21= 2) >>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:40= 4) >>>>> at >>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.ja= va:410) >>>>> at >>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.j= ava:582) >>>>> >>>>> >>>>> And while it is no problem at the beginning of the reduce process, >>>>> where this happens only on a few nodes and rarely, it becomes crucial= as the >>>>> progress rises. The reason for this (afaik from reading articles), is= that >>>>> there are memory or file handle problems. I addressed the memory prob= lem by >>>>> conitiously purging the map of outdated elements evry 5 million proce= ssed >>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit i= n the >>>>> shell tells me it is 400000000). >>>>> >>>>> Anyway i am still running into those mortbay errors and i start to >>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By= pure >>>>> naive math it should be: >>>>> i explicily assigned 10GB memory to each JVM on each node and set >>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops >>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large datastru= ctures >>>>> cause the GC to throw exceptions). This would naively make 18 slave m= achines >>>>> with 10GB each resulting in an overall memory of 180GB - three times = as much >>>>> as needed... i would think. So if the Partitioner distributes them ju= st >>>>> about equally to all nodes i should not run into any errors, do i? >>>>> >>>>> Can anybody help me with this issue? >>>>> >>>>> Best regards, >>>>> Elmar >>>>> >>>>> >>>> >>>> >>> >>> >> >> >