Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of
 admin@nikore.net)
MIME-Version: 1.0
In-Reply-To: 
 <CAH29n6Nfv8LH_bdCaLOHSFbgcUFV8gESq2jK=3Fv=nA+aN2xvg@mail.gmail.com>
References: 
 <CA+HR8ZNDWMd_hp+heBW9OKv1gA1xT8j3CTyTJbDzw3R4DHtUVg@mail.gmail.com>
 <CAH29n6Nfv8LH_bdCaLOHSFbgcUFV8gESq2jK=3Fv=nA+aN2xvg@mail.gmail.com>
From: Matt Christiansen <admin@nikore.net>
Date: Fri, 9 Aug 2013 11:56:03 -0700
Message-ID: 
 <CA+HR8ZNxGbGiUSJd5Zh+R78OWhwk8dGwG05B1UObsYdO9S82OA@mail.gmail.com>
Subject: Re: Crunch Errors while running jobs on YARN
To: user@crunch.apache.org
Content-Type: text/plain; charset=ISO-8859-1

It seems like the job stops if I run it with hadoop jar but not if I
run it with java -cp then again there could be something else im
missing.

It would be great if it could be fixed as I don't like random errors
poping up in my jobs and nor will our QA. I'll try patching crunch
with the ticket you listed above and see if that takes care of the
issue.

On Thu, Aug 8, 2013 at 5:56 AM, Josh Wills <jwills@cloudera.com> wrote:
> Hey Matt,
>
> It looks like this is the problem:
> https://issues.apache.org/jira/browse/MAPREDUCE-4442
>
> Is it causing the job to fail? We could add a similar workaround to the one
> described in the JIRA.
>
> J
>
>
> On Wed, Aug 7, 2013 at 5:37 PM, Matt Christiansen <admin@nikore.net> wrote:
>>
>> Hey guys, im getting this error while running a crunch job on our YARN
>> cluster. It doesn't seem to be serious buts generating a lot of QA
>> questions:
>>
>>
>> ERROR exec.MRExecutor: Exception thrown fetching job counters for
>> stage: com.rr.crunch.ads.KeyValuePairsAggregator:
>>
>> Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1)
>> java.io.IOException
>>      at
>> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
>>      at
>> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337)
>>      at
>> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476)
>>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)
>>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)
>>      at java.security.AccessController.doPrivileged(Native Method)
>>      at javax.security.auth.Subject.doAs(Subject.java:415)
>>      at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>      at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)
>>      at
>> org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69)
>>      at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)
>>      at
>> com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79)
>>      at
>> com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63)
>>
>> This is with crunch 0.5.0 but I have tried 0.7.0 and get the same
>> general error (no message about it being job counters but same strack
>> trace and in the code its at the job.getJob().getCounters() part) I
>> have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any
>> one had come acrossed this before or had any idea
>
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills