Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 734EDD14E for ; Wed, 31 Oct 2012 08:57:18 +0000 (UTC) Received: (qmail 73449 invoked by uid 500); 31 Oct 2012 08:57:18 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 73387 invoked by uid 500); 31 Oct 2012 08:57:18 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 73148 invoked by uid 99); 31 Oct 2012 08:57:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Oct 2012 08:57:15 +0000 Date: Wed, 31 Oct 2012 08:57:15 +0000 (UTC) From: "zhangjianlin (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: <1524395470.49599.1351673835618.JavaMail.jiratomcat@arcas> In-Reply-To: <1301795354.49581.1351673112085.JavaMail.jiratomcat@arcas> Subject: [jira] [Updated] (MAPREDUCE-4759) java.io.IOException: File too large MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-4759?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangjianlin updated MAPREDUCE-4759: ------------------------------------ Description:=20 when running mr job.one of cluster lost tasktracker some times=E3=80=82 see the hadoop-root-tasktracker-xxx.out =EF=BC=9A java.io.IOException: File too large at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:260) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:31= 6) at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRolling= FileAppender.java:359) at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java= :251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppe= nders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.jav= a:133) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(Task= Tracker.java:2972) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.jav= a:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.j= ava:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandle= r.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.j= ava:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.j= ava:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java= :417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(Contex= tHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.j= ava:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.ja= va:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(H= ttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoi= nt.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadP= ool.java:522) see files=20 was: when running mr job.one of cluster lost tasktracker some times=E3=80=82 see the hadoop-root-tasktracker-xxx.out =EF=BC=9A java.io.IOException: File too large at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:260) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:31= 6) at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRolling= FileAppender.java:359) at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java= :251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppe= nders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.jav= a:133) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(Task= Tracker.java:2972) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.jav= a:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.j= ava:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandle= r.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.j= ava:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.j= ava:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java= :417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(Contex= tHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.j= ava:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.ja= va:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(H= ttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoi= nt.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadP= ool.java:522) [thread 1097128256 also had an error] the Java Runtime Environment: # # SIGSEGV (0xb) at pc=3D0x00002adafa2df63c, pid=3D20204, tid=3D1101338944 # # JRE version: 6.0_18-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd= 64 ) # Problematic frame: # V [libjvm.so+0x62263c] # # An error report file with more information is saved as: # /usr/local/hadoop-0.20.2/hs_err_pid20204.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # see /usr/local/hadoop-0.20.2/hs_err_pid20204.log=EF=BC=9A VM Arguments: jvm_args: -Xmx5000m -Dhadoop.log.dir=3D/usr/local/hadoop-0.20.2/bin/../logs= -Dhadoop.log.file=3Dhadoop-root-tasktracker-t0928.log -Dhadoop.home.dir=3D= /usr/local/hadoop-0.20.2/bin/.. -Dhadoop.id.str=3Droot -Dhadoop.root.logger= =3DINFO,DRFA -Djava.library.path=3D/usr/local/hadoop-0.20.2/bin/../lib/nati= ve/Linux-amd64-64 -Dhadoop.policy.file=3Dhadoop-policy.xml=20 java_command: org.apache.hadoop.mapred.TaskTracker Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=3D/usr/java CLASSPATH=3D/usr/local/hadoop-0.20.2/bin/../conf:/usr/java/lib/tools.jar:/u= sr/local/hadoop-0.20.2/bin/..:/usr/local/hadoop-0.20.2/bin/../hadoop-0.20.2= -core.jar:/usr/local/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/usr/loca= l/hadoop-0.20.2/bin/../lib/commons-codec-1.3.jar:/usr/local/hadoop-0.20.2/b= in/../lib/commons-el-1.0.jar:/usr/local/hadoop-0.20.2/bin/../lib/commons-ht= tpclient-3.0.1.jar:/usr/local/hadoop-0.20.2/bin/../lib/commons-io-1.4.jar:/= usr/local/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/usr/local/had= oop-0.20.2/bin/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop-0.20.= 2/bin/../lib/commons-net-1.4.1.jar:/usr/local/hadoop-0.20.2/bin/../lib/core= -3.1.1.jar:/usr/local/hadoop-0.20.2/bin/../lib/hbase-0.90.3.jar:/usr/local/= hadoop-0.20.2/bin/../lib/hbase-0.90.3-tests.jar:/usr/local/hadoop-0.20.2/bi= n/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop-0.20.2/bin/../lib/jasper-com= piler-5.5.12.jar:/usr/local/hadoop-0.20.2/bin/../lib/jasper-runtime-5.5.12.= jar:/usr/local/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop-= 0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop-0.20.2/bin/../lib/jett= y-util-6.1.14.jar:/usr/local/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/usr/= local/hadoop-0.20.2/bin/../lib/kfs-0.2.2.jar:/usr/local/hadoop-0.20.2/bin/.= ./lib/log4j-1.2.15.jar:/usr/local/hadoop-0.20.2/bin/../lib/mail.jar:/usr/lo= cal/hadoop-0.20.2/bin/../lib/mockito-all-1.8.0.jar:/usr/local/hadoop-0.20.2= /bin/../lib/mysql-connector-java-5.1.6-bin.jar:/usr/local/hadoop-0.20.2/bin= /../lib/oro-2.0.8.jar:/usr/local/hadoop-0.20.2/bin/../lib/servlet-api-2.5-6= .1.14.jar:/usr/local/hadoop-0.20.2/bin/../lib/slf4j-api-1.4.3.jar:/usr/loca= l/hadoop-0.20.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop-0.20.2= /bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop-0.20.2/bin/../lib/zookeeper-3= .3.2.jar:/usr/local/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local= /hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-api-2.1.jar PATH=3D/usr/kerberos/sbin:/usr/kerberos/bin::/usr/java/bin:/usr/java/jre/bi= n:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/h= adoop-0.20.2/bin:/usr/local/mysql/bin:/usr/local/bin:/usr/local/zookeeper-3= .3.3/bin:/usr/local/hbase-0.90.3/bin:/usr/local/hive-0.7.1/bin:/home/root1/= bin LD_LIBRARY_PATH=3D/usr/java/jre/lib/amd64/server:/usr/java/jre/lib/amd64:/u= sr/java/jre/../lib/amd64 SHELL=3D/bin/bash Signal Handlers: SIGSEGV: [libjvm.so+0x70f1a0], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x100000= 04 SIGBUS: [libjvm.so+0x70f1a0], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x1000000= 4 SIGFPE: [libjvm.so+0x5d7f70], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x1000000= 4 SIGPIPE: [libjvm.so+0x5d7f70], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x100000= 04 SIGXFSZ: [libjvm.so+0x5d7f70], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x100000= 04 SIGILL: [libjvm.so+0x5d7f70], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x1000000= 4 SIGUSR1: SIG_DFL, sa_mask[0]=3D0x00000000, sa_flags=3D0x00000000 SIGUSR2: [libjvm.so+0x5da790], sa_mask[0]=3D0x00000000, sa_flags=3D0x100000= 04 SIGHUP: SIG_IGN, sa_mask[0]=3D0x00000000, sa_flags=3D0x00000000 SIGINT: SIG_IGN, sa_mask[0]=3D0x00000000, sa_flags=3D0x00000000 SIGTERM: [libjvm.so+0x5da4e0], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x100000= 04 SIGQUIT: [libjvm.so+0x5da4e0], sa_mask[0]=3D0x7ffbfeff, sa_flags=3D0x100000= 04 --------------- S Y S T E M --------------- OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga) uname:Linux 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 libc:glibc 2.5 NPTL 2.5=20 rlimit: STACK 10240k, CORE 0k, NPROC 73728, NOFILE 8192, AS infinity load average:6.77 4.99 2.31 CPU:total 8 (8 cores per cpu, 2 threads per core) family 6 model 26 steppin= g 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, = ht Memory: 4k page, physical 8165800k(1082892k free), swap 20972848k(20454204k= free) vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for linux-amd64 JRE (= 1.6.0_18-b07), built on Dec 17 2009 13:42:22 by "java_re" with gcc 3.2.2 (S= uSE Linux) time: Wed Oct 31 10:58:41 2012 elapsed time: 80860 seconds =20 > java.io.IOException: File too large > ----------------------------------- > > Key: MAPREDUCE-4759 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4759 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller > Affects Versions: 0.20.2 > Reporter: zhangjianlin > Priority: Critical > Labels: hadoop > Fix For: 0.20.2 > > Attachments: hs_err_pid20204.log > > > when running mr job.one of cluster lost tasktracker some times=E3=80=82 > see the hadoop-root-tasktracker-xxx.out =EF=BC=9A > java.io.IOException: File too large > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) > at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:27= 2) > at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) > at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) > at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) > at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58= ) > at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:= 316) > at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRolli= ngFileAppender.java:359) > at org.apache.log4j.WriterAppender.append(WriterAppender.java:160= ) > at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.ja= va:251) > at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAp= penders(AppenderAttachableImpl.java:66) > at org.apache.log4j.Category.callAppenders(Category.java:206) > at org.apache.log4j.Category.forcedLog(Category.java:391) > at org.apache.log4j.Category.log(Category.java:856) > at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.j= ava:133) > at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(Ta= skTracker.java:2972) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.j= ava:502) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler= .java:363) > at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHand= ler.java:216) > at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler= .java:181) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler= .java:766) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.ja= va:417) > at org.mortbay.jetty.handler.ContextHandlerCollection.handle(Cont= extHandlerCollection.java:230) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper= .java:152) > at org.mortbay.jetty.Server.handle(Server.java:324) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.= java:534) > at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete= (HttpConnection.java:864) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:20= 7) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:40= 3) > at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndP= oint.java:409) > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThrea= dPool.java:522) > see files=20 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira