Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of shahab.yunus@gmail.com
 designates 209.85.215.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEo-6+RH_TpuQLtbj9w45ji+7OeX5-cOP7+LH0HmGa81+Dmmcw@mail.gmail.com>
References: 
 <CAEo-6+RH_TpuQLtbj9w45ji+7OeX5-cOP7+LH0HmGa81+Dmmcw@mail.gmail.com>
Date: Fri, 22 Aug 2014 20:39:34 -0400
Message-ID: 
 <CAEo-6+T1LcOCTx=PWhaCgCqS7obttSA8KEmzZAq+ansTjQ7e5Q@mail.gmail.com>
Subject: Re: job.getCounters returns null in Yarn-based job
From: Shahab Yunus <shahab.yunus@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e013d173a154f0305014130e7

--089e013d173a154f0305014130e7
Content-Type: text/plain; charset=UTF-8

For those who are interested, this got resolved.

The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.

The job finishes now and I am able to print and get counters as well.

One minor thing that now the job history UI does not show the history with
the error message that max counter increased.

Regards,
Shahab


On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <shahab.yunus@gmail.com>
wrote:

> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff?  I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>

--089e013d173a154f0305014130e7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">For those who are interested, this got resolved.<div><br><=
/div><div>The issue was that I was creating more counters than what was con=
figured in the settings.</div><div>I upped mapreduce.job.counters.max prope=
rty to a larger number. The default was 120.</div>
<div><br></div><div>The job finishes now and I am able to print and get cou=
nters as well.</div><div><br></div><div>One minor thing that now the job hi=
story UI does not show the history with the error message that max counter =
increased.</div>
<div><br></div><div>Regards,</div><div>Shahab</div></div><div class=3D"gmai=
l_extra"><br><br><div class=3D"gmail_quote">On Fri, Aug 22, 2014 at 7:59 AM=
, Shahab Yunus <span dir=3D"ltr">&lt;<a href=3D"mailto:shahab.yunus@gmail.c=
om" target=3D"_blank">shahab.yunus@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello.<div><br></div><div>I=
 am trying to access custom counters that I have created in an mapreduce jo=
b on Yarn.</div>
<div><br></div><div>After job.waitForCompletion(true) call, I try to do job=
.getCounters() but I get a null.</div>
<div><br></div><div>This only happens if I run a heavy job meaning a) a lot=
 of data and b) lot of reducers.</div><div><br></div><div>E.g. for 10millio=
n records with 20 reducers on a 10 node cluster it works. But on 60million =
records with 70 reducers on a 10 node cluster it doesn&#39;t.=C2=A0</div>

<div><br></div><div>The job itself competes successfully.</div><div><br></d=
iv><div>I did see the following related JIRAs. But the first one is for old=
 version, pre-Yarn and those properties I think are not valid anymore.</div=
>

<div><br></div><div>The second one does not seem to provide a solution? I t=
ried using the suggested trick on the client side but no success there eith=
er.</div><div><br></div><div><a href=3D"https://issues.apache.org/jira/brow=
se/MAPREDUCE-1920" target=3D"_blank">MAPREDUCE-1920</a></div>

<div><a href=3D"https://issues.apache.org/jira/browse/MAPREDUCE-4442" targe=
t=3D"_blank">MAPREDUCE-4442</a></div><div><br></div><div>Please advise, how=
 can I retrieve my custom counters after jobs completion. Am I missing some=
thing? Do I need to configure some job history stuff? =C2=A0I do a mention =
of ATS as well but I don&#39;t know how much that is applicable here.</div>

<div><br></div><div>Thanks a lot.<br></div><div><br></div><div>My version i=
s:=C2=A02.3.0-cdh5.1.0</div><div><br></div><div>Regards,</div><div>Shahab</=
div></div>
</blockquote></div><br></div>

--089e013d173a154f0305014130e7--