Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ACF87118F8 for ; Mon, 14 Apr 2014 12:27:32 +0000 (UTC) Received: (qmail 77856 invoked by uid 500); 14 Apr 2014 12:27:23 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 77747 invoked by uid 500); 14 Apr 2014 12:27:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77569 invoked by uid 99); 14 Apr 2014 12:27:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Apr 2014 12:27:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dingweiqiong@gmail.com designates 209.85.216.169 as permitted sender) Received: from [209.85.216.169] (HELO mail-qc0-f169.google.com) (209.85.216.169) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Apr 2014 12:27:16 +0000 Received: by mail-qc0-f169.google.com with SMTP id i17so8800094qcy.28 for ; Mon, 14 Apr 2014 05:26:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=yUPaWwGD88IiY6Nb0uMysgDaqFtq157u3GoRQkaWaxg=; b=qQWi+59OB12I1+6fz37IlN21JT7M9gCpB4LB28/I6Sx+rTLZ/0m/uJsIit44dd8ovY 1x4Idr2OK6dh3Msvs8DhvZ3qtsijfC8DecjJm4R5P6ldBLsWu1cjIA2p/YAnIR3HekDF OgFpikiINvQam4t5DVJmpAnwjChLvbWRvwJhe8XuCpXAwujV+lo7IBv/M4U2tZYUhuCA h8P1k63ROVV37gGLcYg+iTV1P/unBS9SnqTVsqY+SoxwYj5Ly/Y2cqSFSPTNKo2mx8Ke TsCPefQLswvSJpR2R+QByhP+OY1P7vf4peegpFeeC0HyPqdYhhOdl/Qj38t9qGEmGkiW CqTA== MIME-Version: 1.0 X-Received: by 10.140.106.195 with SMTP id e61mr47169481qgf.66.1397478416053; Mon, 14 Apr 2014 05:26:56 -0700 (PDT) Received: by 10.140.102.242 with HTTP; Mon, 14 Apr 2014 05:26:56 -0700 (PDT) In-Reply-To: References: Date: Mon, 14 Apr 2014 20:26:56 +0800 Message-ID: Subject: Re: Exception in hadoop jobtracker OOM From: dingweiqiong To: Alysa Liu Content-Type: multipart/alternative; boundary=001a113a7a7c91fc7e04f6ffcc0e X-Virus-Checked: Checked by ClamAV on apache.org --001a113a7a7c91fc7e04f6ffcc0e Content-Type: text/plain; charset=UTF-8 unsubscribe 2014-04-12 23:02 GMT+08:00 Viswanathan J : > Hi, > > I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without > any issues. > > Suddenly I got the below error in Jobtracker and jobs are failed to run. > > Is this issue in JT or TT or Jetty issue? > > 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION > java.lang.OutOfMemoryError: Java heap space at java.util.HashMap. > newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet. > iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153) > at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127) > at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at > sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at > sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at > org.mortbay.io.nio. SelectorManager$SelectSet. > doSelect(SelectorManager.java:498) at org.mortbay.io.nio. > SelectorManager.doSelect( SelectorManager.java:192) at > org.mortbay.jetty.nio. SelectChannelConnector.accept( > SelectChannelConnector.java:124) at org.mortbay.jetty. > AbstractConnector$Acceptor. run(AbstractConnector.java:708) at > org.mortbay.thread. QueuedThreadPool$PoolThread. > run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR > org.apache.hadoop.security. UserGroupInformation: > PriviledgedActionException as:root cause:java.io.IOException: > java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR > org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java > heap space at java.lang.String.toCharArray( String.java:2725) at > sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at > sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at > sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at > sun.misc.URLClassPath. getResource(URLClassPath.java:169) at > java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security. > AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. > findClass(URLClassLoader.java:190) at java.lang.ClassLoader. > loadClass(ClassLoader.java:306) at sun.misc.Launcher$ > AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader. > loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred. > jobtracker_jsp._jspService( jobtracker_jsp.java:508) at > org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at > javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at > org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at > org.mortbay.jetty.servlet. ServletHandler$CachedChain. > doFilter(ServletHandler.java:1221) at org.apache.hadoop.http. > HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at > org.mortbay.jetty.servlet. ServletHandler$CachedChain. > doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet. > ServletHandler.handle(ServletHandler.java:399) at > org.mortbay.jetty.security. SecurityHandler.handle( > SecurityHandler.java:216) at org.mortbay.jetty.servlet. > SessionHandler.handle( SessionHandler.java:182) at > org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766) > at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450) > at org.mortbay.jetty.handler. ContextHandlerCollection.handle( > ContextHandlerCollection.java:230) at org.mortbay.jetty.handler. > HandlerWrapper.handle( HandlerWrapper.java:152) at > org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty. > HttpConnection.handleRequest( HttpConnection.java:542) at > org.mortbay.jetty. HttpConnection$RequestHandler. > headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser. > parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser. > parseAvailable(HttpParser.java:212) at > org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at > org.mortbay.io.nio. SelectChannelEndPoint.run( > SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR > org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE > canonical subfolder. java.lang.OutOfMemoryError: Java heap space > > Please help as I'm facing in production. > --001a113a7a7c91fc7e04f6ffcc0e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
unsubscribe


2014-04-12 23:02 GMT+08:00 Viswanathan J <ja= yamviswanathan@gmail.com>:

Hi,

I'm using Hadoop v1.2.1 and it is running fine so long(3 months) wit= hout any issues.

Suddenly I got the below error in Jobtracker and jobs are failed to run.=

Is this issue in JT or TT or Jetty issue?

2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOf= MemoryError: Java heap space at java.util.HashMap. newKeyIterator(HashMap.j= ava:840) at java.util.HashMap$KeySet. iterator(HashMap.java:874) at java.ut= il.HashSet.iterator( HashSet.java:153) at sun.nio.ch.SelectorImpl. processD= eregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl. doSe= lect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl. lockAndDoSelect= (SelectorImpl.java:69) at sun.nio.ch.SelectorImpl. select(SelectorImpl.java= :80) at org.mortbay.io.nio. SelectorManager$SelectSet. doSelect(SelectorMan= ager.java:498) at org.mortbay.io.nio. SelectorManager.doSelect( SelectorMan= ager.java:192) at org.mortbay.jetty.nio. SelectChannelConnector.accept( Sel= ectChannelConnector.java:124) at org.mortbay.jetty. AbstractConnector$Accep= tor. run(AbstractConnector.java:708) at org.mortbay.thread. QueuedThreadPoo= l$PoolThread. run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR = org.apache.hadoop.security. UserGroupInformation: PriviledgedActionExceptio= n as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap = space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.= jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharA= rray( String.java:2725) at sun.net.www.ParseUtil. encodePath(ParseUtil.java= :86) at sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:6= 74) at sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) = at sun.misc.URLClassPath. getResource(URLClassPath.java:169) at java.net.UR= LClassLoader$1.run( URLClassLoader.java:194) at java.security. AccessContro= ller.doPrivileged(Native Method) at java.net.URLClassLoader. findClass(URLC= lassLoader.java:190) at java.lang.ClassLoader. loadClass(ClassLoader.java:3= 06) at sun.misc.Launcher$ AppClassLoader.loadClass(Launcher.java:301) at ja= va.lang.ClassLoader. loadClass(ClassLoader.java:247) at org.apache.hadoop.m= apred. jobtracker_jsp._jspService( jobtracker_jsp.java:508) at org.apache.j= asper.runtime. HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.ht= tp.HttpServlet.service( HttpServlet.java:820) at org.mortbay.jetty.servlet.= ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.= ServletHandler$CachedChain. doFilter(ServletHandler.java:1221) at org.apac= he.hadoop.http. HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914= ) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(Servle= tHandler.java:1212) at org.mortbay.jetty.servlet. ServletHandler.handle(Ser= vletHandler.java:399) at org.mortbay.jetty.security. SecurityHandler.handle= ( SecurityHandler.java:216) at org.mortbay.jetty.servlet. SessionHandler.ha= ndle( SessionHandler.java:182) at org.mortbay.jetty.handler. ContextHandler= .handle( ContextHandler.java:766) at org.mortbay.jetty.webapp. WebAppContex= t.handle( WebAppContext.java:450) at org.mortbay.jetty.handler. ContextHand= lerCollection.handle( ContextHandlerCollection.java:230) at org.mortbay.jet= ty.handler. HandlerWrapper.handle( HandlerWrapper.java:152) at org.mortbay.= jetty.Server. handle(Server.java:326) at org.mortbay.jetty. HttpConnection.= handleRequest( HttpConnection.java:542) at org.mortbay.jetty. HttpConnectio= n$RequestHandler. headerComplete(HttpConnection.java:928) at org.mortbay.je= tty.HttpParser. parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpPar= ser. parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnectio= n.handle( HttpConnection.java:404) at org.mortbay.io.nio. SelectChannelEndP= oint.run( SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org= .apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canon= ical subfolder. java.lang.OutOfMemoryError: Java heap space

Please help as I'm facing in production.


--001a113a7a7c91fc7e04f6ffcc0e--