hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent conf across JobClient, MRAppMaster, and YarnChild
Date Sun, 04 May 2014 02:22:15 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988890#comment-13988890
] 

Gera Shegalov commented on MAPREDUCE-5875:
------------------------------------------

[~sandyr], there is definitely a substantial overlap. Sorry that I missed it which would save
some time. However  MAPREDUCE-5856 does not seem to address uberized jobs /AM and jobs submitted
without JobClient. See the unit tests. I'll link to the JIRA's for further discussion. 

> Make Counter limits consistent conf across JobClient, MRAppMaster, and YarnChild
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5875
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, client, task
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5875.v01.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by {{org.apache.hadoop.mapreduce.counters.Limits}}
are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max counters/groups is
only increased when needed. With the current Limits implementation relying on static constants,
it's going to be challenging for tools that submit jobs concurrently (e.g. via class loading
isolation).
> The patch that I am uploading is not perfect but demonstrates the issue. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message