Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of jwills@cloudera.com
 designates 209.85.216.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+HR8ZNDWMd_hp+heBW9OKv1gA1xT8j3CTyTJbDzw3R4DHtUVg@mail.gmail.com>
References: 
 <CA+HR8ZNDWMd_hp+heBW9OKv1gA1xT8j3CTyTJbDzw3R4DHtUVg@mail.gmail.com>
From: Josh Wills <jwills@cloudera.com>
Date: Thu, 8 Aug 2013 05:56:32 -0700
Message-ID: 
 <CAH29n6Nfv8LH_bdCaLOHSFbgcUFV8gESq2jK=3Fv=nA+aN2xvg@mail.gmail.com>
Subject: Re: Crunch Errors while running jobs on YARN
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=14dae9c09bf42eccf404e36f3137

--14dae9c09bf42eccf404e36f3137
Content-Type: text/plain; charset=ISO-8859-1

Hey Matt,

It looks like this is the problem:
https://issues.apache.org/jira/browse/MAPREDUCE-4442

Is it causing the job to fail? We could add a similar workaround to the one
described in the JIRA.

J


On Wed, Aug 7, 2013 at 5:37 PM, Matt Christiansen <admin@nikore.net> wrote:

> Hey guys, im getting this error while running a crunch job on our YARN
> cluster. It doesn't seem to be serious buts generating a lot of QA
> questions:
>
>
> ERROR exec.MRExecutor: Exception thrown fetching job counters for
> stage: com.rr.crunch.ads.KeyValuePairsAggregator:
>
> Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1)
> java.io.IOException
>      at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
>      at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337)
>      at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476)
>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)
>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)
>      at java.security.AccessController.doPrivileged(Native Method)
>      at javax.security.auth.Subject.doAs(Subject.java:415)
>      at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>      at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)
>      at
> org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69)
>      at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)
>      at
> com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79)
>      at
> com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63)
>
> This is with crunch 0.5.0 but I have tried 0.7.0 and get the same
> general error (no message about it being job counters but same strack
> trace and in the code its at the job.getJob().getCounters() part) I
> have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any
> one had come acrossed this before or had any idea
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

--14dae9c09bf42eccf404e36f3137
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey Matt,<div><br></div><div>It looks like this is the pro=
blem:=A0<a href=3D"https://issues.apache.org/jira/browse/MAPREDUCE-4442">ht=
tps://issues.apache.org/jira/browse/MAPREDUCE-4442</a></div><div><br></div>=
<div>

Is it causing the job to fail? We could add a similar workaround to the one=
 described in the JIRA.</div><div><br></div><div>J</div></div><div class=3D=
"gmail_extra"><br><br><div class=3D"gmail_quote">On Wed, Aug 7, 2013 at 5:3=
7 PM, Matt Christiansen <span dir=3D"ltr">&lt;<a href=3D"mailto:admin@nikor=
e.net" target=3D"_blank">admin@nikore.net</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hey guys, im getting this error while runnin=
g a crunch job on our YARN<br>
cluster. It doesn&#39;t seem to be serious buts generating a lot of QA<br>
questions:<br>
<br>
<br>
ERROR exec.MRExecutor: Exception thrown fetching job counters for<br>
stage: com.rr.crunch.ads.KeyValuePairsAggregator:<br>
Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-dis=
tinct+S1+SeqFile(/tmp/crunch-1725623982/p1)<br>
java.io.IOException<br>
=A0 =A0 =A0at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientS=
erviceDelegate.java:317)<br>
=A0 =A0 =A0at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters=
(ClientServiceDelegate.java:337)<br>
=A0 =A0 =A0at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner=
.java:476)<br>
=A0 =A0 =A0at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)<br>
=A0 =A0 =A0at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)<br>
=A0 =A0 =A0at java.security.AccessController.doPrivileged(Native Method)<br=
>
=A0 =A0 =A0at javax.security.auth.Subject.doAs(Subject.java:415)<br>
=A0 =A0 =A0at org.apache.hadoop.security.UserGroupInformation.doAs(UserGrou=
pInformation.java:1408)<br>
=A0 =A0 =A0at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)<br>
=A0 =A0 =A0at org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.=
java:69)<br>
=A0 =A0 =A0at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)=
<br>
=A0 =A0 =A0at com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePai=
rsAggregator.java:79)<br>
=A0 =A0 =A0at com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsA=
ggregator.java:63)<br>
<br>
This is with crunch 0.5.0 but I have tried 0.7.0 and get the same<br>
general error (no message about it being job counters but same strack<br>
trace and in the code its at the job.getJob().getCounters() part) I<br>
have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any<br>
one had come acrossed this before or had any idea<br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div>Directo=
r of Data Science</div><div><a href=3D"http://www.cloudera.com" target=3D"_=
blank">Cloudera</a></div><div>Twitter: <a href=3D"http://twitter.com/josh_w=
ills" target=3D"_blank">@josh_wills</a></div>


</div>

--14dae9c09bf42eccf404e36f3137--