hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Age Mooij (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-2088) TaskGraphServlet throws ArrayIndexOutOfBoundsException when progress % > 100%
Date Fri, 24 Sep 2010 12:51:33 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Age Mooij updated MAPREDUCE-2088:

    Attachment: MAPREDUCE-2088-JobTracker.png

I attached two screenshots, one for the job overview as it currently stands and one for the
reducers progress page as it currently stands.

> TaskGraphServlet throws ArrayIndexOutOfBoundsException when progress % > 100%
> -----------------------------------------------------------------------------
>                 Key: MAPREDUCE-2088
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2088
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.20.2
>         Environment: Hadoop 0.20.2, r911707 running on a linux cluster with 4 worker
nodes and LZO compressed map output
>            Reporter: Age Mooij
>         Attachments: MAPREDUCE-2088-JobTracker.png, MAPREDUCE-2088-ReducerProgress.png
> I'm currently running a map reduce job in which the reducer inserts data into HBase.
It is a long running job dealing with hundreds of GB of map output data and 9 out of 10 reducers
are currently showing 110+% progress (i.e. the total job reduce progress is 100% but clickig
through to the reducer list shows reducers with percentages over 100%). 
> The number of reduce input records is 4,299,991,005
> While keeping an eye on it through the job detail page of the JobTracker web gui, the
Jobtracker started generating exceptions every 30 seconds which turned out to be generated
by the TaskGrapServlet that displays reducer progress. The stacktrace also showed up in the
web gui in the area that normally holds the reducer progress graph. 30 seconds is of course
the refresh rate of the status page.
> I guess the root cause here is progress percentages being greater than 100%. I found
issue HADOOP-5210 which reports a cause for progress crossing 100% but it is marked as fixed
in 0.20.1 and I'm running 0.20.2.
> One thing that might impact the progress percentages is the fact that we LZO compress
our map outputs. For the above job uncompressed output ("Map output bytes") was 2,462,412,228,874
which compressed to 694,405,054,632 (FILE_BYTES_WRITTEN/map).
> Here's the exception:
> 2010-09-23 18:03:05,484 ERROR org.mortbay.log: /taskgraph
> java.lang.ArrayIndexOutOfBoundsException: 3
> 	at org.apache.hadoop.mapred.TaskGraphServlet.getReduceAvarageProgresses(TaskGraphServlet.java:199)
> 	at org.apache.hadoop.mapred.TaskGraphServlet.doGet(TaskGraphServlet.java:131)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> 	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.Server.handle(Server.java:324)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> 	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message