hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: job.getCounters returns null in Yarn-based job
Date Sat, 23 Aug 2014 00:39:34 GMT
For those who are interested, this got resolved.

The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.

The job finishes now and I am able to print and get counters as well.

One minor thing that now the job history UI does not show the history with
the error message that max counter increased.

Regards,
Shahab


On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <shahab.yunus@gmail.com>
wrote:

> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff?  I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>

Mime
View raw message