Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7B1E11860B for ; Fri, 4 Mar 2016 01:10:41 +0000 (UTC) Received: (qmail 14732 invoked by uid 500); 4 Mar 2016 01:10:41 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 14645 invoked by uid 500); 4 Mar 2016 01:10:41 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 14405 invoked by uid 99); 4 Mar 2016 01:10:41 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2016 01:10:41 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id DC7522C1F5A for ; Fri, 4 Mar 2016 01:10:40 +0000 (UTC) Date: Fri, 4 Mar 2016 01:10:40 +0000 (UTC) From: "zhihai xu (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179080#comment-15179080 ] zhihai xu commented on MAPREDUCE-6622: -------------------------------------- Thanks for the confirmation [~rchiang]! I will back port the patch to 2.6.5 and 2.7.3 branch after waiting for several days if no one objects. > Add capability to set JHS job cache to a task-based limit > --------------------------------------------------------- > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver > Affects Versions: 2.7.2 > Reporter: Ray Chiang > Assignee: Ray Chiang > Priority: Critical > Labels: supportability > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6622.001.patch, MAPREDUCE-6622.002.patch, MAPREDUCE-6622.003.patch, MAPREDUCE-6622.004.patch, MAPREDUCE-6622.005.patch, MAPREDUCE-6622.006.patch, MAPREDUCE-6622.007.patch, MAPREDUCE-6622.008.patch, MAPREDUCE-6622.009.patch, MAPREDUCE-6622.010.patch, MAPREDUCE-6622.011.patch, MAPREDUCE-6622.012.patch, MAPREDUCE-6622.013.patch, MAPREDUCE-6622.014.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs can be of varying size. This is generally not a problem when the jobs sizes are uniform or small, but when the job sizes can be very large (say greater than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and spend all its time in GC. However, since the cache is holding on to all the jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the cache and since the total number of tasks loaded is directly proportional to the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)