hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
Date Fri, 29 Jan 2016 15:26:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123592#comment-15123592
] 

Jason Lowe commented on MAPREDUCE-6622:
---------------------------------------

Thanks for the patch, Ray!

The documentation for the existing property for limiting based on job count needs to be updated
to mention it is ignored if the new property is set.

Seems like it would be straightforward to pull out old jobs until we've freed up enough tasks
to pay for the job trying to be added to the cache.  Now we have a background thread with
a new property to configure how often it cleans -- does the cache blow way out of proportion
if we don't clean up fast enough (e.g.: history server gets programmatically hammered for
many jobs)? Adding yet another guava dependency isn't appealing unless really necessary. 
If we are sticking with the guava cache, is it essential to call cleanUp in a background thread
or won't this cleanup automatically happen as new jobs are loaded into the cache?



> Add capability to set JHS job cache to a task-based limit
> ---------------------------------------------------------
>
>                 Key: MAPREDUCE-6622
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobhistoryserver
>    Affects Versions: 2.7.2
>            Reporter: Ray Chiang
>            Assignee: Ray Chiang
>              Labels: supportability
>         Attachments: MAPREDUCE-6622.001.patch
>
>
> When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs can be
of varying size.  This is generally not a problem when the jobs sizes are uniform or small,
but when the job sizes can be very large (say greater than 250k tasks), then the JHS heap
size can grow tremendously.
> In cases, where multiple jobs are very large, then the JHS can lock up and spend all
its time in GC.  However, since the cache is holding on to all the jobs, not much heap space
can be freed up.
> By setting a property that sets a cap on the number of tasks allowed in the cache and
since the total number of tasks loaded is directly proportional to the amount of heap used,
this should help prevent the JHS from locking up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message