hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ray Chiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6222) HistoryServer Hangs Processing Large Jobs
Date Thu, 30 Apr 2015 17:39:07 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ray Chiang updated MAPREDUCE-6222:
    Status: Open  (was: Patch Available)

Found some log pollution due to this patch.  I sent an email to mapreduce-dev, but no replies


In Dispatcher#service(), I see this comment:

      // TODO: support args converted from /path/:arg1/...
      dest.action.invoke(controller, (Object[]) null);

Right now, I've made some changes in MAPREDUCE-6222 that seems to trigger exceptions at this
TODO.  Can someone give me a clearer idea about what sort of processing should occur at the
point of this TODO?

On a related note, is there any additional/better documentation about the various org.apache.hadoop.yarn.webapp
and org.apache.hadoop.yarn.webapp.view classes?  There are a few things I'm trying to figure
out and stepping through with a debugger gets painful at times.

Any information is appreciated.  Thanks.

> HistoryServer Hangs Processing Large Jobs
> -----------------------------------------
>                 Key: MAPREDUCE-6222
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6222
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Andrew Johnson
>            Assignee: Ray Chiang
>         Attachments: JHS New Display Top.png, JHS Original Display Top.png, MAPREDUCE-6222.001.patch,
MAPREDUCE-6222.002.patch, MAPREDUCE-6222.003.patch, MAPREDUCE-6222.005.patch, head.jhist,
> I'm encountering an issue with the Mapreduce HistoryServer processing the history files
for large jobs.  This has come up several times with for jobs with around 60000 total tasks.
 When the HistoryServer loads the .jhist file from HDFS for a job of that size (which is usually
around 500 Mb), the HistoryServer's CPU usage spiked and the UI became unresponsive.  After
about 10 minutes I restarted the HistoryServer and it was behaving normally again.
> The cluster is running CDH 5.3 (2.5.0-cdh5.3.0).  I've attached the output of jstack
from a time this was occurring.

This message was sent by Atlassian JIRA

View raw message