hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Morel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
Date Wed, 03 Jun 2015 14:56:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570953#comment-14570953

David Morel commented on MAPREDUCE-3971:

Wondering as well, or wether I should open a separate ticket. What I'm noticing when accessing
 /ws/v1/history/mapreduce/jobs/{jobid} for jobs having over 100 Ktasks is a sharp rise in
MemHeapUsedM and a server that becomes unresponsive, even though the heap is set pretty high.
I suspect this is due to parsing the whole jhist file, which I've seen instances of over 900MB.
As a result, to extract stats that I need, I had to write my own external script to parse
the jhist files retrieved from HDFS. Moreover, i noticed some bad interactions when using
oozie (tasks seen as KILLED even though the log says they're SUCCEEDED), making the history
server a SPOF in my architecture...

> Job History web services need to have limits on the number of items they can return.
> ------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3971
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.2
>            Reporter: Robert Joseph Evans
> The Job History web services canput a very large load on the job history server.  We
should put in a limit on the number of entries that can be returned by the web service, and
also add in the ability to modify the starting location in the list, so that all entries can
still be downlaoded.  Just not all at once.

This message was sent by Atlassian JIRA

View raw message