hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
Date Mon, 31 Jul 2017 21:38:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108008#comment-16108008
] 

Hudson commented on MAPREDUCE-5875:
-----------------------------------

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12080 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12080/])
Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, (junping_du: rev
fbb7d6bcbb887ce52ab1e9d5a1fed67a7f8a4be8)
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java


> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5875
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, client, task
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>             Fix For: 2.9.0, 3.0.0-alpha1
>
>         Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch, MAPREDUCE-5875.v03.patch,
MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch, MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch,
MAPREDUCE-5875.v08.patch, MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by {{org.apache.hadoop.mapreduce.counters.Limits}}
are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max counters/groups is
only increased when needed. With the current Limits implementation relying on static constants,
it's going to be challenging for tools that submit jobs concurrently  without resorting to
class loading isolation.
> The patch that I am uploading is not perfect but demonstrates the issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message