hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
Date Thu, 01 Mar 2012 21:28:01 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220378#comment-13220378
] 

Robert Joseph Evans commented on MAPREDUCE-3944:
------------------------------------------------

I agree with Tom on this.  I would also like to be able to limit the total number of entries
that can be returned, but we have the issue with ordering here like Sid said.  I am going
to try to set up a stress test for this so I can try out several fixes to see what happens.
 There are really two different goals to this JIRA.  The most important one is that even if
someone is hitting the web service a lot, the history server should still be usable.  The
second part is to optimize the web service calls so that they reduce the load on the system.
 I think that the locking is the primary thing for fixing the first part, and I will put in
the filtering changes too.  Once I have those working I will do some profiling and see what
else looks like hot spots.

                
> JobHistory web services are slower then the UI and can easly overload the JH
> ----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3944
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.1, 0.23.2
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>            Priority: Blocker
>
> When our first customer started using the Job History web services today the History
Server ground to a halt.  We found 250 Jetty threads stuck on the following stack trace.
> {noformat}
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
>         - waiting to lock <0x00002aaab364ba60> (a org.apache.hadoop.mapreduce.v2.hs.JobHistory)
>         at org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done to improve
its performance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message