Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3DFD810A7E for ; Tue, 24 Mar 2015 05:19:54 +0000 (UTC) Received: (qmail 44685 invoked by uid 500); 24 Mar 2015 05:19:54 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 44620 invoked by uid 500); 24 Mar 2015 05:19:54 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 44605 invoked by uid 99); 24 Mar 2015 05:19:53 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2015 05:19:53 +0000 Date: Tue, 24 Mar 2015 05:19:53 +0000 (UTC) From: "Gera Shegalov (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377308#comment-14377308 ] Gera Shegalov commented on MAPREDUCE-5875: ------------------------------------------ [~rkanter], Oh my, we messed something up with this JIRA. Both MAPREDUCE-6288 and MAPREDUCE-6286 are broken because we dropped the logic in v08 during rebase to v09. > Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild > --------------------------------------------------------------------------- > > Key: MAPREDUCE-5875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, client, task > Affects Versions: 2.4.0 > Reporter: Gera Shegalov > Assignee: Gera Shegalov > Fix For: 2.7.0 > > Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch, MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch, MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch, MAPREDUCE-5875.v09.patch > > > Currently, counter limits "mapreduce.job.counters.*" handled by {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas it's taken into account in YarnChild. > It would be good to make the Limits job-configurable, such that max counters/groups is only increased when needed. With the current Limits implementation relying on static constants, it's going to be challenging for tools that submit jobs concurrently without resorting to class loading isolation. > The patch that I am uploading is not perfect but demonstrates the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)