Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <23000466.383981285332693377.JavaMail.jira@thor>
Date: Fri, 24 Sep 2010 08:51:33 -0400 (EDT)
From: "Age Mooij (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Updated: (MAPREDUCE-2088) TaskGraphServlet throws
 ArrayIndexOutOfBoundsException when progress % > 100%
In-Reply-To: <27267879.383641285332213768.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Age Mooij updated MAPREDUCE-2088:
---------------------------------

    Attachment: MAPREDUCE-2088-JobTracker.png
                MAPREDUCE-2088-ReducerProgress.png

I attached two screenshots, one for the job overview as it currently stands and one for the reducers progress page as it currently stands.

> TaskGraphServlet throws ArrayIndexOutOfBoundsException when progress % > 100%
> -----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2088
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2088
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.20.2
>         Environment: Hadoop 0.20.2, r911707 running on a linux cluster with 4 worker nodes and LZO compressed map output
>            Reporter: Age Mooij
>         Attachments: MAPREDUCE-2088-JobTracker.png, MAPREDUCE-2088-ReducerProgress.png
>
>
> I'm currently running a map reduce job in which the reducer inserts data into HBase. It is a long running job dealing with hundreds of GB of map output data and 9 out of 10 reducers are currently showing 110+% progress (i.e. the total job reduce progress is 100% but clickig through to the reducer list shows reducers with percentages over 100%). 
> The number of reduce input records is 4,299,991,005
> While keeping an eye on it through the job detail page of the JobTracker web gui, the Jobtracker started generating exceptions every 30 seconds which turned out to be generated by the TaskGrapServlet that displays reducer progress. The stacktrace also showed up in the web gui in the area that normally holds the reducer progress graph. 30 seconds is of course the refresh rate of the status page.
> I guess the root cause here is progress percentages being greater than 100%. I found issue HADOOP-5210 which reports a cause for progress crossing 100% but it is marked as fixed in 0.20.1 and I'm running 0.20.2.
> One thing that might impact the progress percentages is the fact that we LZO compress our map outputs. For the above job uncompressed output ("Map output bytes") was 2,462,412,228,874 which compressed to 694,405,054,632 (FILE_BYTES_WRITTEN/map).
> Here's the exception:
> 2010-09-23 18:03:05,484 ERROR org.mortbay.log: /taskgraph
> java.lang.ArrayIndexOutOfBoundsException: 3
> 	at org.apache.hadoop.mapred.TaskGraphServlet.getReduceAvarageProgresses(TaskGraphServlet.java:199)
> 	at org.apache.hadoop.mapred.TaskGraphServlet.doGet(TaskGraphServlet.java:131)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> 	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.Server.handle(Server.java:324)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> 	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.