Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EF088D4A5 for ; Wed, 5 Sep 2012 13:53:03 +0000 (UTC) Received: (qmail 24574 invoked by uid 500); 5 Sep 2012 13:52:58 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 24448 invoked by uid 500); 5 Sep 2012 13:52:57 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 24441 invoked by uid 99); 5 Sep 2012 13:52:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 13:52:57 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=FSL_RCVD_USER,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [141.51.167.101] (HELO gundel.cs.uni-kassel.de) (141.51.167.101) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 13:52:51 +0000 Received: from localhost (localhost [127.0.0.1]) by gundel.cs.uni-kassel.de (Postfix) with ESMTP id 4B3DE20C9A1 for ; Wed, 5 Sep 2012 15:52:29 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at gundel.cs.uni-kassel.de Received: from gundel.cs.uni-kassel.de ([127.0.0.1]) by localhost (gundel.cs.uni-kassel.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9JMJtslVINVU for ; Wed, 5 Sep 2012 15:52:19 +0200 (CEST) Received: from [141.51.123.117] (fudd.cs.uni-kassel.de [141.51.123.117]) by gundel.cs.uni-kassel.de (Postfix) with ESMTPSA id 7893B20C99E for ; Wed, 5 Sep 2012 15:52:19 +0200 (CEST) Message-ID: <50475922.4040706@cs.uni-kassel.de> Date: Wed, 05 Sep 2012 15:52:34 +0200 From: =?ISO-8859-1?Q?Bj=F6rn-Elmar_Macek?= User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120824 Thunderbird/15.0 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: Re: mortbay, huge files and the ulimit References: <503BAA39.9070607@gmail.com> <503E046D.1080904@gmail.com> <503E1EDC.80501@cs.uni-kassel.de> <503F400A.8060702@cs.uni-kassel.de> <5040A956.5090401@cs.uni-kassel.de> <50473DE9.8020008@cs.uni-kassel.de> <50473F69.9030905@cs.uni-kassel.de> <50474A20.9020806@cs.uni-kassel.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Hi Vasco, i think both heuristics dont imply anything about the sense or nonsense of the program: The first version of my program used the "values" as "keys", so i could accomplish some sorting. But this also meant, that since each value is unique my iterator in the reduce method was also always of size 1. So i had to store possible joinpartners in a reduce-function-external variable and i decided to put it as a membervariable, which i regularly cleaned of outdated data. This data structure grew with every call of the reduce function and with this the size of non-garbage-collectable data, while the code in my class didnt do too much: since i used a hashmap here, there was only very few time-consuming traversing needed and the execution time of my code was most probably smaller than the execution time of the GC since a certain treshold. In fact: it is easy to create a program that does meaningfull stuff and still trigger the 98%-2% rule and let the GC throw an exception. My 2 cents, Elmar Am 05.09.2012 15:32, schrieb Vasco Visser: > Now that I think about it I wonder about oracle's description. > > [...] if more than 98% of the total time is spent in garbage collection [...] > > I took that to mean 98% of CPU time for the application, but it can > also mean that the GC is active for 98% percent of wall clock time. > > If it is the former it implies the application isn't doing useful > stuff anymore (e.g., keeping a connection open). If it is the latter I > don't know if it possible to say anything about how much the program > is still doing useful stuff. > > > > On Wed, Sep 5, 2012 at 2:48 PM, Bj�rn-Elmar Macek > wrote: >> Hi Vasco, >> >> thank you for your help! >> >> I can try to add the limit again (i currently have it turned off for all >> Java processes spawned by Hadoop). Also i do not have any persistent >> (member-) variables, that could store things that use alot of data: at the >> very moment i only got 2 local variables in the reduce-method, that could >> get a little larger - but those should be gc'ed easily, shouldnt they? >> >> Thank you for your thoughts again. I will pay with the GC a little. If i >> know the reason, i'll let you know! >> Best regards, >> Elmar >> >> >> Am 05.09.2012 14:38, schrieb Vasco Visser: >> >>> Just a guess, but could it simply be memory issue on reducer? >>> >>> In your adjusted program, maybe try running without >>> -UseGCOverheadLimit and see if you still got OOM errors. >>> >>> From the sun website: >>> The parallel collector will throw an OutOfMemoryError if too much time >>> is being spent in garbage collection: if more than 98% of the total >>> time is spent in garbage collection and less than 2% of the heap is >>> recovered, an OutOfMemoryError will be thrown. This feature is >>> designed to prevent applications from running for an extended period >>> of time while making little or no progress because the heap is too >>> small. If necessary, this feature can be disabled by adding the option >>> -XX:-UseGCOverheadLimit to the command line. >>> >>> So basically if you get a GC overhead limit exceeded OOM error it >>> means that your app is doing nothing but garbage collection, the vm is >>> fighting against running out of memory. If that happens it might >>> result in a timeout of the connection fetching the map data (i don't >>> know if it can result in timeout, but I can imagine it could). >>> >>> Also, note that 63 GB on disk is probably going to be inflated in >>> memory. So in general you cant say that 60 GB on disk will need 60GB >>> of mem. Actually, some people us a rule of thumb to do x4 to get >>> approx mem requirement. >>> >>> Just some ideas, not really a solution but maybe it helps you further. >>> >>> On Wed, Sep 5, 2012 at 2:02 PM, Bj�rn-Elmar Macek >>> wrote: >>>> Excuse me: in my last code section was some old code included. Here is it >>>> again stripped of deprecated code: >>>> >>>> >>>> package uni.kassel.macek.rtprep; >>>> >>>> >>>> >>>> import gnu.trove.iterator.TIntIterator; >>>> import gnu.trove.map.hash.TIntObjectHashMap; >>>> import gnu.trove.set.TIntSet; >>>> >>>> import java.io.IOException; >>>> import java.util.ArrayList; >>>> import java.util.Calendar; >>>> import java.util.Iterator; >>>> import java.util.List; >>>> import java.util.StringTokenizer; >>>> >>>> import org.apache.hadoop.conf.Configuration; >>>> import org.apache.hadoop.fs.Path; >>>> import org.apache.hadoop.io.IntWritable; >>>> import org.apache.hadoop.io.LongWritable; >>>> import org.apache.hadoop.io.Text; >>>> import org.apache.hadoop.mapred.MapReduceBase; >>>> import org.apache.hadoop.mapred.OutputCollector; >>>> import org.apache.hadoop.mapred.Reporter; >>>> import org.apache.hadoop.mapreduce.Job; >>>> import org.apache.hadoop.mapreduce.Mapper; >>>> import org.apache.hadoop.mapreduce.Reducer; >>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; >>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; >>>> import org.apache.hadoop.util.GenericOptionsParser; >>>> >>>> public class RetweetApplication { >>>> >>>> public static class RetweetMapper1 extends Mapper>>> Text, >>>> Text> { >>>> >>>> public void map(Object key, Text value, >>>> Context context) >>>> throws IOException { >>>> >>>> String line = value.toString(); >>>> >>>> Tweet aTweet = new Tweet(line); >>>> if(aTweet.getMention()!=null && >>>> aTweet.getMention().length()>0){ >>>> /*aTweet.setRT(0); >>>> output.collect(new Text(aTweet.toString()), new >>>> Text(aTweet.toString()));*/ >>>> aTweet.setRT(1); >>>> try { >>>> context.write(new Text(aTweet.getMention()), new >>>> Text(aTweet.toString())); >>>> } catch (InterruptedException e) { >>>> // TODO Auto-generated catch block >>>> e.printStackTrace(); >>>> } >>>> } >>>> aTweet.setRT(0); >>>> try { >>>> context.write(new Text(aTweet.getAuthor()), new >>>> Text(aTweet.toString())); >>>> } catch (InterruptedException e) { >>>> // TODO Auto-generated catch block >>>> e.printStackTrace(); >>>> } >>>> >>>> } >>>> } >>>> >>>> public static class RetweetReducer1 extends Reducer>>> Text> >>>> { >>>> >>>> public void reduce(Text atweet, Iterator tweets, >>>> Context context) >>>> throws IOException { >>>> List originals = new ArrayList(); >>>> List retweets = new ArrayList(); >>>> while(tweets.hasNext()){ >>>> Tweet aTweet = new Tweet(tweets.toString()); >>>> originals.add(aTweet); >>>> if(aTweet.getRT()>0) retweets.add(aTweet); >>>> } >>>> >>>> for(Tweet originalTweet : originals){ >>>> for(Tweet aRetweet : retweets){ >>>> if(aRetweet.isRetweeted(originalTweet)) >>>> { >>>> aRetweet.setReactionTime(originalTweet.getDate()); >>>> try { >>>> context.write( >>>> atweet, >>>> new Text(aRetweet.toString())); >>>> } catch (InterruptedException e) { >>>> // TODO Auto-generated catch block >>>> e.printStackTrace(); >>>> } >>>> } >>>> } >>>> } >>>> } >>>> } >>>> >>>> public static void main(String[] args) throws Exception { >>>> Configuration conf = new Configuration(); >>>> String[] otherArgs = new GenericOptionsParser(conf, >>>> args).getRemainingArgs(); >>>> if (otherArgs.length != 2) { >>>> System.err.println("Usage: wordcount "); >>>> System.exit(2); >>>> } >>>> Job job = new Job(conf, "word count"); >>>> >>>> job.setJarByClass(RetweetApplication.class); >>>> job.setMapperClass(RetweetMapper1.class); >>>> job.setReducerClass(RetweetReducer1.class); >>>> job.setOutputKeyClass(Text.class); >>>> job.setOutputValueClass(Text.class); >>>> FileInputFormat.addInputPath(job, new Path(otherArgs[0])); >>>> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); >>>> System.exit(job.waitForCompletion(true) ? 0 : 1); >>>> } >>>> } >>>> Am 05.09.2012 13:56, schrieb Bj�rn-Elmar Macek: >>>> >>>>> Hello again, >>>>> >>>>> i just wanted to keep you updated, in case anybody reads this and is >>>>> interested in the mortbay-issue: >>>>> I applied the wordcount example to a big input file and everything >>>>> worked >>>>> fine. I watched the TT logs all the time and no single mortbay-eof >>>>> exception >>>>> occurred. The only possible explanation for me seemed to be, that my >>>>> code >>>>> somehow produces the exceptions. So i started to keep my code as close >>>>> to >>>>> the wordcount example as possible: >>>>> >>>>> * first getting rid of any members within the reducer class, that may >>>>> take >>>>> lots of memory. >>>>> * using the Context class instead of OutputCollector (in Mapper and >>>>> Reducer) >>>>> * removing any Comparator (and restructuring the code so it just >>>>> calculates the same as with the sorting) >>>>> * using standart HashPartitioner >>>>> >>>>> Especially the first 2 points made my job succeed in the end, ALTHOUGH >>>>> the >>>>> problem of "n-1 mortbay-exception-throwing TTs" still remains: one node >>>>> seems to complete all the work while the others are busy with their >>>>> exceptions (at least from what i can see in the log). >>>>> >>>>> I REALLY dont know what i can do, to make this job run through like the >>>>> wordcount example. I dont see any big difference between my program and >>>>> the >>>>> example anymore. >>>>> >>>>> I added the code here: >>>>> >>>>> *Please* help me here. >>>>> >>>>> Best regards, >>>>> Elmar >>>>> >>>>> >>>>> I added the code here: >>>>> ######################################## >>>>> package uni.kassel.macek.rtprep; >>>>> >>>>> import java.util.Date; >>>>> >>>>> import org.apache.hadoop.conf.Configuration; >>>>> import org.apache.hadoop.conf.Configured; >>>>> import org.apache.hadoop.fs.Path; >>>>> import org.apache.hadoop.io.Text; >>>>> import org.apache.hadoop.mapred.FileInputFormat; >>>>> import org.apache.hadoop.mapred.FileOutputFormat; >>>>> import org.apache.hadoop.mapred.JobClient; >>>>> import org.apache.hadoop.mapred.JobConf; >>>>> import org.apache.hadoop.mapred.TextOutputFormat; >>>>> import org.apache.hadoop.util.Tool; >>>>> import org.apache.hadoop.util.ToolRunner; >>>>> >>>>> public class RetweetApplication extends Configured implements Tool{ >>>>> >>>>> public static int DEBUG = -1; >>>>> private Configuration jobConf; >>>>> >>>>> //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT"; >>>>> //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new >>>>> Date()).toString(); >>>>> >>>>> public static void main(String[] args) { >>>>> int res=-1; >>>>> try { >>>>> res = ToolRunner.run(new Configuration(), new >>>>> RetweetApplication(), args); >>>>> } catch (Exception e) { >>>>> // TODO Auto-generated catch block >>>>> e.printStackTrace(); >>>>> } >>>>> System.exit(res); >>>>> >>>>> } >>>>> >>>>> public void setConf(Configuration conf) { >>>>> jobConf = conf; >>>>> >>>>> } >>>>> >>>>> public Configuration getConf() { >>>>> return jobConf; >>>>> } >>>>> >>>>> public int run(String[] args) throws Exception { >>>>> JobClient client = new JobClient(); >>>>> JobConf conf = new JobConf(RetweetApplication.class); >>>>> >>>>> >>>>> if(DEBUG > 0){ >>>>> conf.set("mapred.job.tracker", "local"); >>>>> conf.set("fs.default.name", "file:///"); >>>>> conf.set("dfs.replication", "1"); >>>>> } >>>>> >>>>> //conf.set("mapred.child.java.opts","-Xmx3000m >>>>> -XX:+UseCompressedOops -XX:-UseGCOverheadLimit"); >>>>> conf.set("mapred.child.ulimit","100000000"); >>>>> >>>>> FileInputFormat.setInputPaths(conf, new Path(args[0])); >>>>> FileOutputFormat.setOutputPath(conf, new Path(args[1])); >>>>> >>>>> >>>>> //conf.setOutputKeyClass(Text.class); >>>>> //conf.setOutputValueClass(Text.class); >>>>> conf.setMapOutputKeyClass(Text.class); >>>>> conf.setMapOutputValueClass(Text.class); >>>>> >>>>> conf.setMapperClass(RetweetMapper.class); >>>>> //conf.setPartitionerClass(TweetPartitioner.class); >>>>> >>>>> //conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class); >>>>> >>>>> >>>>> //conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class); >>>>> conf.setReducerClass(RetweetReducer.class); >>>>> >>>>> conf.setOutputFormat(TextOutputFormat.class); >>>>> >>>>> client.setConf(conf); >>>>> try { >>>>> JobClient.runJob(conf); >>>>> } catch (Exception e) { >>>>> e.printStackTrace(); >>>>> } >>>>> return 0; >>>>> } >>>>> } >>>>> */ >>>>> >>>>> package uni.kassel.macek.rtprep; >>>>> >>>>> >>>>> >>>>> import gnu.trove.iterator.TIntIterator; >>>>> import gnu.trove.map.hash.TIntObjectHashMap; >>>>> import gnu.trove.set.TIntSet; >>>>> >>>>> import java.io.IOException; >>>>> import java.util.ArrayList; >>>>> import java.util.Calendar; >>>>> import java.util.Iterator; >>>>> import java.util.List; >>>>> import java.util.StringTokenizer; >>>>> >>>>> import org.apache.hadoop.conf.Configuration; >>>>> import org.apache.hadoop.fs.Path; >>>>> import org.apache.hadoop.io.IntWritable; >>>>> import org.apache.hadoop.io.LongWritable; >>>>> import org.apache.hadoop.io.Text; >>>>> import org.apache.hadoop.mapred.MapReduceBase; >>>>> import org.apache.hadoop.mapred.OutputCollector; >>>>> import org.apache.hadoop.mapred.Reporter; >>>>> import org.apache.hadoop.mapreduce.Job; >>>>> import org.apache.hadoop.mapreduce.Mapper; >>>>> import org.apache.hadoop.mapreduce.Reducer; >>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; >>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; >>>>> import org.apache.hadoop.util.GenericOptionsParser; >>>>> >>>>> public class RetweetApplication { >>>>> >>>>> public static class RetweetMapper1 extends Mapper>>>> Text, >>>>> Text> { >>>>> >>>>> public void map(Object key, Text value, >>>>> Context context) >>>>> throws IOException { >>>>> >>>>> String line = value.toString(); >>>>> >>>>> Tweet aTweet = new Tweet(line); >>>>> if(aTweet.getMention()!=null && >>>>> aTweet.getMention().length()>0){ >>>>> /*aTweet.setRT(0); >>>>> output.collect(new Text(aTweet.toString()), new >>>>> Text(aTweet.toString()));*/ >>>>> aTweet.setRT(1); >>>>> try { >>>>> context.write(new Text(aTweet.getMention()), new >>>>> Text(aTweet.toString())); >>>>> } catch (InterruptedException e) { >>>>> // TODO Auto-generated catch block >>>>> e.printStackTrace(); >>>>> } >>>>> } >>>>> aTweet.setRT(0); >>>>> try { >>>>> context.write(new Text(aTweet.getAuthor()), new >>>>> Text(aTweet.toString())); >>>>> } catch (InterruptedException e) { >>>>> // TODO Auto-generated catch block >>>>> e.printStackTrace(); >>>>> } >>>>> >>>>> } >>>>> } >>>>> >>>>> public static class RetweetReducer1 extends Reducer>>>> Text> { >>>>> >>>>> public void reduce(Text atweet, Iterator tweets, >>>>> Context context) >>>>> throws IOException { >>>>> List originals = new ArrayList(); >>>>> List retweets = new ArrayList(); >>>>> while(tweets.hasNext()){ >>>>> Tweet aTweet = new Tweet(tweets.toString()); >>>>> originals.add(aTweet); >>>>> if(aTweet.getRT()>0) retweets.add(aTweet); >>>>> } >>>>> >>>>> for(Tweet originalTweet : originals){ >>>>> for(Tweet aRetweet : retweets){ >>>>> if(aRetweet.isRetweeted(originalTweet)) >>>>> { >>>>> aRetweet.setReactionTime(originalTweet.getDate()); >>>>> try { >>>>> context.write( >>>>> atweet, >>>>> new Text(aRetweet.toString())); >>>>> } catch (InterruptedException e) { >>>>> // TODO Auto-generated catch block >>>>> e.printStackTrace(); >>>>> } >>>>> } >>>>> } >>>>> } >>>>> } >>>>> } >>>>> >>>>> public static void main(String[] args) throws Exception { >>>>> Configuration conf = new Configuration(); >>>>> String[] otherArgs = new GenericOptionsParser(conf, >>>>> args).getRemainingArgs(); >>>>> if (otherArgs.length != 2) { >>>>> System.err.println("Usage: wordcount "); >>>>> System.exit(2); >>>>> } >>>>> Job job = new Job(conf, "tweets"); >>>>> job.setJarByClass(RetweetApplication.class); >>>>> job.setMapperClass(RetweetMapper1.class); >>>>> job.setReducerClass(RetweetReducer1.class); >>>>> job.setOutputKeyClass(Text.class); >>>>> job.setOutputValueClass(Text.class); >>>>> FileInputFormat.addInputPath(job, new Path(otherArgs[0])); >>>>> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); >>>>> System.exit(job.waitForCompletion(true) ? 0 : 1); >>>>> } >>>>> } >>>>> Am 31.08.2012 14:08, schrieb Bj�rn-Elmar Macek: >>>>>> Hi all, >>>>>> >>>>>> i got some more information: >>>>>> >>>>>> It is always only one TT, where mortbay errors do not occur: all other >>>>>> slaves fail relatively fast after the reduce process started: maybe it >>>>>> is a >>>>>> comparator issue (since the operation is MAPRED_SHUFFLE) >>>>>> And it has nothing to do with the local configuration of the server, >>>>>> since >>>>>> 1) if i rerun the job on the same set of servers, the ONE working >>>>>> differs >>>>>> from one job to another. >>>>>> 2) the same happens even when i am using a completely different set of >>>>>> servers. >>>>>> >>>>>> Who is capable to solve this unsolved mystery? :D >>>>>> >>>>>> Am 30.08.2012 12:27, schrieb Bj�rn-Elmar Macek: >>>>>>> Hi, >>>>>>> >>>>>>> is there anybody, who knows more about this issue: it has already been >>>>>>> recently marked here: >>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5 >>>>>>> >>>>>>> I really want to do something about it, if i knew how... i tried so >>>>>>> many >>>>>>> different setup parameters and JVM options and nothing did the >>>>>>> trick... >>>>>>> >>>>>>> It was opened 2009 and gets still updated (2012): I mean - this does >>>>>>> NOT >>>>>>> look OK, does it? The whole TT seems like a big time waster: >>>>>>> >>>>>>> 2012-08-30 12:05:22,582 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39586, bytes: 146086410, op: MAPRED_SHUFFLE, >>>>>>> cliID: >>>>>>> attempt_201208301039_0001_m_000189_0, duration: 5456122852 >>>>>>> 2012-08-30 12:05:56,275 INFO org.mortbay.log: >>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>>>>> injecting >>>>>>> delay2 times >>>>>>> 2012-08-30 12:05:56,276 INFO org.mortbay.log: >>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>>>>> recreating selector 2 times, canceled keys 70 times >>>>>>> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: >>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 >>>>>>> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:08:39,236 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39614, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>>>>> attempt_201208301039_0001_m_000204_0, duration: 3209410 >>>>>>> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput >>>>>>> java.lang.IllegalStateException: Committed >>>>>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>>>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> 2012-08-30 12:10:27,109 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39618, bytes: 145730308, op: MAPRED_SHUFFLE, >>>>>>> cliID: >>>>>>> attempt_201208301039_0001_m_000204_0, duration: 5686096517 >>>>>>> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: >>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 >>>>>>> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:14:29,740 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39648, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>>>>> attempt_201208301039_0001_m_000236_0, duration: 3746212 >>>>>>> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput >>>>>>> java.lang.IllegalStateException: Committed >>>>>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>>>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> 2012-08-30 12:16:28,503 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39652, bytes: 145506420, op: MAPRED_SHUFFLE, >>>>>>> cliID: >>>>>>> attempt_201208301039_0001_m_000236_0, duration: 7793391393 >>>>>>> 2012-08-30 12:16:59,421 INFO org.mortbay.log: >>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>>>>> injecting >>>>>>> delay1 times >>>>>>> 2012-08-30 12:16:59,422 INFO org.mortbay.log: >>>>>>> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - >>>>>>> recreating selector 1 times, canceled keys 69 times >>>>>>> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: >>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 >>>>>>> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed : >>>>>>> org.mortbay.jetty.EofException >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> Caused by: java.io.IOException: Broken pipe >>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) >>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:60) >>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) >>>>>>> at >>>>>>> org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221) >>>>>>> at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725) >>>>>>> ... 27 more >>>>>>> >>>>>>> 2012-08-30 12:20:01,205 INFO >>>>>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: >>>>>>> 141.51.205.130:35497, >>>>>>> dest: 141.51.205.126:39683, bytes: 65536, op: MAPRED_SHUFFLE, cliID: >>>>>>> attempt_201208301039_0001_m_000241_0, duration: 5405031 >>>>>>> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput >>>>>>> java.lang.IllegalStateException: Committed >>>>>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>>>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>> at >>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>> at >>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>> at >>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>> at >>>>>>> >>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>> >>>>>>> Am 29.08.12 15:53, schrieb Bj�rn-Elmar Macek: >>>>>>>> Hi there, >>>>>>>> >>>>>>>> i am currently running a job where i selfjoin a 63 gigabyte big csv >>>>>>>> file on 20 physically distinct nodes with 15GB each: >>>>>>>> >>>>>>>> While the mapping works just fine and is low cost, the reducer does >>>>>>>> the >>>>>>>> main work: holding a hashmap with elements to join with and finding >>>>>>>> join >>>>>>>> tuples for evry incoming key-value-pair. >>>>>>>> >>>>>>>> The jobs works perfectly on small files with 2 gigabytes, but starts >>>>>>>> to >>>>>>>> get "unstable" as the file size goes up: this becomes evident with a >>>>>>>> look >>>>>>>> into the tasktracker's logs saying: >>>>>>>> >>>>>>>> ERROR org.mortbay.log: /mapOutput >>>>>>>> java.lang.IllegalStateException: Committed >>>>>>>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) >>>>>>>> at org.mortbay.jetty.Response.sendError(Response.java:240) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945) >>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>>>>>> at >>>>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>>>>>> at org.mortbay.jetty.Server.handle(Server.java:326) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>>>>>> at >>>>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>>>>>> at >>>>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>>>>>> at >>>>>>>> >>>>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>>>>>> >>>>>>>> >>>>>>>> And while it is no problem at the beginning of the reduce process, >>>>>>>> where this happens only on a few nodes and rarely, it becomes crucial >>>>>>>> as the >>>>>>>> progress rises. The reason for this (afaik from reading articles), is >>>>>>>> that >>>>>>>> there are memory or file handle problems. I addressed the memory >>>>>>>> problem by >>>>>>>> conitiously purging the map of outdated elements evry 5 million >>>>>>>> processed >>>>>>>> key-value-pairs. And i set mapred.child.ulimit to 100000000 (ulimit >>>>>>>> in the >>>>>>>> shell tells me it is 400000000). >>>>>>>> >>>>>>>> Anyway i am still running into those mortbay errors and i start to >>>>>>>> wonder, if hadoop can manage the job with this algorithmn anyways. By >>>>>>>> pure >>>>>>>> naive math it should be: >>>>>>>> i explicily assigned 10GB memory to each JVM on each node and set >>>>>>>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops >>>>>>>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large >>>>>>>> datastructures >>>>>>>> cause the GC to throw exceptions). This would naively make 18 slave >>>>>>>> machines >>>>>>>> with 10GB each resulting in an overall memory of 180GB - three times >>>>>>>> as much >>>>>>>> as needed... i would think. So if the Partitioner distributes them >>>>>>>> just >>>>>>>> about equally to all nodes i should not run into any errors, do i? >>>>>>>> >>>>>>>> Can anybody help me with this issue? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Elmar >>>>>>>> >>>>>>>>