Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3EDD0DD4A for ; Wed, 23 Jan 2013 10:52:03 +0000 (UTC) Received: (qmail 2335 invoked by uid 500); 23 Jan 2013 10:51:58 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 2114 invoked by uid 500); 23 Jan 2013 10:51:58 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 15041 invoked by uid 99); 23 Jan 2013 09:40:24 -0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=DNS_FROM_AHBL_RHSBL,FREEMAIL_ENVFROM_END_DIGIT,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of s2323@land.ru designates 62.141.94.205 as permitted sender) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qip.ru; s=dkim; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:Date:Subject:To:Sender:From; bh=wVdbLTED0Hj+u4tl8X/XFuTPHcovTwVM//dRlqN7eR0=; b=N6hz4jd8u2Ge4WI3fcXzYlxtFjlo4QBlTAN/cIP5gr4I0SWiVMxCse7GggajnKyYoZMq4HdNvORZ14umX40yUxILeNMIexnJHOpj1v3hAxlbGcRAv1FroMJ1wI4mE8PU; From: s2323 Sender: s2323@land.ru To: user@hadoop.apache.org Subject: EOF when Combiner works Date: Wed, 23 Jan 2013 13:39:56 +0400 Message-Id: X-Priority: 3 X-QIP-Sender: 217.174.98.237 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-NoSpam-Exim-Host: 62.141.94.133 X-NoSpam-Exim-Port: 8092 X-NoSpam-Exim-Scanned: Yes X-NoSpam-Exim-Result: OK X-Virus-Checked: Checked by ClamAV on apache.org Hi!=0D=0A=0D=0AWhen I run job with this options:=0D=0A -Dmapred.map.chi= ld.java.opts=3D-Xmx2048M=0D=0A -Dio.sort.mb=3D1424=0D=0A -Dio.sort.rec= ord.percent=3D0.08=0D=0Aall tasks fails on combiner step with:=0D=0A=0D= =0A...=0D=0A2013-01-23 12:20:28,143 INFO org.apache.hadoop.mapred.MapTas= k: io.sort.mb =3D 1424=0D=0A2013-01-23 12:23:03,772 INFO org.apache.hado= op.mapred.MapTask: data buffer =3D 1098974720/1373718448=0D=0A2013-01-23= 12:23:03,772 INFO org.apache.hadoop.mapred.MapTask: record buffer =3D 5= 972689/7465861=0D=0A2013-01-23 12:23:03,790 INFO com.hadoop.compression.= lzo.GPLNativeCodeLoader: Loaded native gpl library=0D=0A2013-01-23 12:23= :03,792 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded &= initialized native-lzo library [hadoop-lzo rev Unknown build revision]= =0D=0A2013-01-23 12:51:47,211 INFO org.apache.hadoop.mapred.MapTask: Spi= lling map output: buffer full=3D true=0D=0A2013-01-23 12:51:47,211 INFO= org.apache.hadoop.mapred.MapTask: bufstart =3D 0; bufend =3D 1098974551= ; bufvoid =3D 1373718448=0D=0A2013-01-23 12:51:47,211 INFO org.apache.ha= doop.mapred.MapTask: kvstart =3D 0; kvend =3D 5661226; length =3D 746586= 1=0D=0A2013-01-23 12:52:01,389 INFO org.apache.hadoop.io.compress.CodecP= ool: Got brand-new compressor=0D=0A2013-01-23 12:52:02,526 INFO org.apac= he.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with ma= pRetainSize=3D-1 and reduceRetainSize=3D-1=0D=0A2013-01-23 12:52:02,528= ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActio= nException as:user (auth:SIMPLE) cause:java.io.IOException: Spill failed= =0D=0A2013-01-23 12:52:02,529 WARN org.apache.hadoop.mapred.Child: Error= running child=0D=0Ajava.io.IOException: Spill failed=0D=0A=09at org.apa= che.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:886)=0D= =0A=09at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTa= sk.java:574)=0D=0A=09at org.apache.hadoop.mapreduce.TaskInputOutputConte= xt.write(TaskInputOutputContext.java:80)=0D=0A=09at my.mapreduce.Statist= icsHostMapper.calculateCounters(StatisticsHostMapper.java:104)=0D=0A=09a= t my.mapreduce.StatisticsHostMapper.map(StatisticsHostMapper.java:116)= =0D=0A=09at my.mapreduce.StatisticsHostMapper.map(StatisticsHostMapper.j= ava:1)=0D=0A=09at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144= )=0D=0A=09at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:= 647)=0D=0A=09at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)= =0D=0A=09at org.apache.hadoop.mapred.Child$4.run(Child.java:266)=0D=0A= =09at java.security.AccessController.doPrivileged(Native Method)=0D=0A= =09at javax.security.auth.Subject.doAs(Subject.java:396)=0D=0A=09at org.= apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja= va:1278)=0D=0A=09at org.apache.hadoop.mapred.Child.main(Child.java:260)= =0D=0ACaused by: java.lang.RuntimeException: next value iterator failed= =0D=0A=09at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next= (ReduceContext.java:166)=0D=0A=09at my.mapreduce.HostCounters.calculate(= HostCounters.java:147)=0D=0A=09at my.mapreduce.StatisticsHostCombiner.re= duce(StatisticsHostCombiner.java:17)=0D=0A=09at my.mapreduce.StatisticsH= ostCombiner.reduce(StatisticsHostCombiner.java:1)=0D=0A=09at org.apache.= hadoop.mapreduce.Reducer.run(Reducer.java:176)=0D=0A=09at org.apache.had= oop.mapred.Task$NewCombinerRunner.combine(Task.java:1445)=0D=0A=09at org= .apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:= 1291)=0D=0A=09at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access= $1800(MapTask.java:712)=0D=0A=09at org.apache.hadoop.mapred.MapTask$MapO= utputBuffer$SpillThread.run(MapTask.java:1199)=0D=0ACaused by: java.io.E= OFException=0D=0A=09at java.io.DataInputStream.readFully(DataInputStream= .java:180)=0D=0A=09at java.io.DataInputStream.readLong(DataInputStream.j= ava:399)=0D=0A=09at my.mapreduce.HostCounters.readFields(HostCounters.ja= va:54)=0D=0A=09at org.apache.hadoop.io.serializer.WritableSerialization$= WritableDeserializer.deserialize(WritableSerialization.java:67)=0D=0A=09= at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeseria= lizer.deserialize(WritableSerialization.java:40)=0D=0A=09at org.apache.h= adoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)=0D=0A= =09at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(Reduc= eContext.java:163)=0D=0A=09... 8 more=0D=0A2013-01-23 12:52:02,578 INFO= org.apache.hadoop.mapred.Task: Runnning cleanup for the task=0D=0A=0D= =0A=0D=0A=0D=0A=0D=0AWhen I run the same task with this options:=0D=0A = -Dmapred.map.child.java.opts=3D-Xmx2048M=0D=0A -Dio.sort.mb=3D1024=0D= =0A -Dio.sort.record.percent=3D0.08=0D=0Aeverything is OK:=0D=0A=0D=0A.= ..=0D=0A2013-01-23 13:01:46,168 INFO org.apache.hadoop.mapred.MapTask: i= o.sort.mb =3D 1024=0D=0A2013-01-23 13:01:47,593 INFO org.apache.hadoop.m= apred.MapTask: data buffer =3D 790273984/987842480=0D=0A2013-01-23 13:01= :47,593 INFO org.apache.hadoop.mapred.MapTask: record buffer =3D 4294967= /5368709=0D=0A2013-01-23 13:01:47,602 INFO com.hadoop.compression.lzo.GP= LNativeCodeLoader: Loaded native gpl library=0D=0A2013-01-23 13:01:47,62= 1 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initia= lized native-lzo library [hadoop-lzo rev Unknown build revision]=0D=0A20= 13-01-23 13:28:42,168 INFO org.apache.hadoop.mapred.MapTask: Spilling ma= p output: buffer full=3D true=0D=0A2013-01-23 13:28:42,169 INFO org.apac= he.hadoop.mapred.MapTask: bufstart =3D 0; bufend =3D 790273835; bufvoid= =3D 987842480=0D=0A2013-01-23 13:28:42,169 INFO org.apache.hadoop.mapre= d.MapTask: kvstart =3D 0; kvend =3D 4072970; length =3D 5368709=0D=0A201= 3-01-23 13:28:56,417 INFO org.apache.hadoop.io.compress.CodecPool: Got b= rand-new compressor=0D=0A2013-01-23 13:29:18,998 INFO org.apache.hadoop.= mapred.MapTask: Finished spill 0=0D=0A...=0D=0A=0D=0A=0D=0APlease help m= e to understand the reason of task fails.