crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Christiansen <ad...@nikore.net>
Subject Re: Crunch Errors while running jobs on YARN
Date Fri, 09 Aug 2013 18:56:03 GMT
It seems like the job stops if I run it with hadoop jar but not if I
run it with java -cp then again there could be something else im
missing.

It would be great if it could be fixed as I don't like random errors
poping up in my jobs and nor will our QA. I'll try patching crunch
with the ticket you listed above and see if that takes care of the
issue.

On Thu, Aug 8, 2013 at 5:56 AM, Josh Wills <jwills@cloudera.com> wrote:
> Hey Matt,
>
> It looks like this is the problem:
> https://issues.apache.org/jira/browse/MAPREDUCE-4442
>
> Is it causing the job to fail? We could add a similar workaround to the one
> described in the JIRA.
>
> J
>
>
> On Wed, Aug 7, 2013 at 5:37 PM, Matt Christiansen <admin@nikore.net> wrote:
>>
>> Hey guys, im getting this error while running a crunch job on our YARN
>> cluster. It doesn't seem to be serious buts generating a lot of QA
>> questions:
>>
>>
>> ERROR exec.MRExecutor: Exception thrown fetching job counters for
>> stage: com.rr.crunch.ads.KeyValuePairsAggregator:
>>
>> Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1)
>> java.io.IOException
>>      at
>> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
>>      at
>> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337)
>>      at
>> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476)
>>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)
>>      at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)
>>      at java.security.AccessController.doPrivileged(Native Method)
>>      at javax.security.auth.Subject.doAs(Subject.java:415)
>>      at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>      at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)
>>      at
>> org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69)
>>      at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)
>>      at
>> com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79)
>>      at
>> com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63)
>>
>> This is with crunch 0.5.0 but I have tried 0.7.0 and get the same
>> general error (no message about it being job counters but same strack
>> trace and in the code its at the job.getJob().getCounters() part) I
>> have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any
>> one had come acrossed this before or had any idea
>
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills

Mime
View raw message