Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Message-ID: <27720420.383631285332213280.JavaMail.jira@thor>
Date: Fri, 24 Sep 2010 08:43:33 -0400 (EDT)
From: "Age Mooij (JIRA)" <jira@apache.org>
To: mapreduce-dev@hadoop.apache.org
Subject: [jira] Created: (MAPREDUCE-2088) TaskGraphServlet throws
 ArrayIndexOutOfBoundsException when progress % > 100%
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

TaskGraphServlet throws ArrayIndexOutOfBoundsException when progress % > 100%
-----------------------------------------------------------------------------

                 Key: MAPREDUCE-2088
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2088
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: jobtracker
    Affects Versions: 0.20.2
         Environment: Hadoop 0.20.2, r911707 running on a linux cluster with 4 worker nodes and LZO compressed map output
            Reporter: Age Mooij


I'm currently running a map reduce job in which the reducer inserts data into HBase. It is a long running job dealing with hundreds of GB of map output data and 9 out of 10 reducers are currently showing 110+% progress (i.e. the total job reduce progress is 100% but clickig through to the reducer list shows reducers with percentages over 100%). 

The number of reduce input records is 4,299,991,005

While keeping an eye on it through the job detail page of the JobTracker web gui, the Jobtracker started generating exceptions every 30 seconds which turned out to be generated by the TaskGrapServlet that displays reducer progress. The stacktrace also showed up in the web gui in the area that normally holds the reducer progress graph. 30 seconds is of course the refresh rate of the status page.

I guess the root cause here is progress percentages being greater than 100%. I found issue HADOOP-5210 which reports a cause for progress crossing 100% but it is marked as fixed in 0.20.1 and I'm running 0.20.2.

One thing that might impact the progress percentages is the fact that we LZO compress our map outputs. For the above job uncompressed output ("Map output bytes") was 2,462,412,228,874 which compressed to 694,405,054,632 (FILE_BYTES_WRITTEN/map).

Here's the exception:

2010-09-23 18:03:05,484 ERROR org.mortbay.log: /taskgraph
java.lang.ArrayIndexOutOfBoundsException: 3
	at org.apache.hadoop.mapred.TaskGraphServlet.getReduceAvarageProgresses(TaskGraphServlet.java:199)
	at org.apache.hadoop.mapred.TaskGraphServlet.doGet(TaskGraphServlet.java:131)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.