crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: Crunch Errors while running jobs on YARN
Date Tue, 13 Aug 2013 22:31:19 GMT
Which version of hadoop-2 are you testing against? Did you start up the mapreduce specific
JobHistoryServer?

Thanks,
+Vinod

On Aug 7, 2013, at 5:37 PM, Matt Christiansen wrote:

> Hey guys, im getting this error while running a crunch job on our YARN
> cluster. It doesn't seem to be serious buts generating a lot of QA
> questions:
> 
> 
> ERROR exec.MRExecutor: Exception thrown fetching job counters for
> stage: com.rr.crunch.ads.KeyValuePairsAggregator:
> Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1)
> java.io.IOException
>     at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
>     at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337)
>     at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476)
>     at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)
>     at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>     at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)
>     at org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69)
>     at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)
>     at com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79)
>     at com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63)
> 
> This is with crunch 0.5.0 but I have tried 0.7.0 and get the same
> general error (no message about it being job counters but same strack
> trace and in the code its at the job.getJob().getCounters() part) I
> have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any
> one had come acrossed this before or had any idea


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message