hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject job.getCounters returns null in Yarn-based job
Date Fri, 22 Aug 2014 11:59:15 GMT

I am trying to access custom counters that I have created in an mapreduce
job on Yarn.

After job.waitForCompletion(true) call, I try to do job.getCounters() but I
get a null.

This only happens if I run a heavy job meaning a) a lot of data and b) lot
of reducers.

E.g. for 10million records with 20 reducers on a 10 node cluster it works.
But on 60million records with 70 reducers on a 10 node cluster it doesn't.

The job itself competes successfully.

I did see the following related JIRAs. But the first one is for old
version, pre-Yarn and those properties I think are not valid anymore.

The second one does not seem to provide a solution? I tried using the
suggested trick on the client side but no success there either.

MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>

Please advise, how can I retrieve my custom counters after jobs completion.
Am I missing something? Do I need to configure some job history stuff?  I
do a mention of ATS as well but I don't know how much that is applicable

Thanks a lot.

My version is: 2.3.0-cdh5.1.0


View raw message