crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Christiansen <ad...@nikore.net>
Subject Re: Crunch Errors while running jobs on YARN
Date Wed, 14 Aug 2013 01:46:29 GMT
I will give that a try; the idea in the MR ticket listed in the
earlier part of the thread did not work for this.



On Tue, Aug 13, 2013 at 5:47 PM, Josh Wills <jwills@cloudera.com> wrote:
> Hey Matt,
>
> I do a build against CDH4.3.0 and post it on Cloudera's local repository for
> my customers:
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/crunch/crunch-core/
>
> Does that work any better?
>
> J
>
>
> On Tue, Aug 13, 2013 at 5:37 PM, Matt Christiansen <admin@nikore.net> wrote:
>>
>> CDH4.3.0 which I believe its 2.0.5-alpha (not sure)
>>
>> Yes the mapreduce specific JobHistoryServer is started and configured.
>> Our other jobs don't seem to have issues just crunch.
>>
>> On Tue, Aug 13, 2013 at 3:31 PM, Vinod Kumar Vavilapalli
>> <vinodkv@hortonworks.com> wrote:
>> > Which version of hadoop-2 are you testing against? Did you start up the
>> > mapreduce specific JobHistoryServer?
>> >
>> > Thanks,
>> > +Vinod
>> >
>> > On Aug 7, 2013, at 5:37 PM, Matt Christiansen wrote:
>> >
>> > Hey guys, im getting this error while running a crunch job on our YARN
>> > cluster. It doesn't seem to be serious buts generating a lot of QA
>> > questions:
>> >
>> >
>> > ERROR exec.MRExecutor: Exception thrown fetching job counters for
>> > stage: com.rr.crunch.ads.KeyValuePairsAggregator:
>> >
>> > Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1)
>> > java.io.IOException
>> >     at
>> >
>> > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
>> >     at
>> >
>> > org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337)
>> >     at
>> > org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476)
>> >     at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757)
>> >     at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754)
>> >     at java.security.AccessController.doPrivileged(Native Method)
>> >     at javax.security.auth.Subject.doAs(Subject.java:415)
>> >     at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> >     at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754)
>> >     at
>> > org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69)
>> >     at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153)
>> >     at
>> >
>> > com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79)
>> >     at
>> >
>> > com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63)
>> >
>> > This is with crunch 0.5.0 but I have tried 0.7.0 and get the same
>> > general error (no message about it being job counters but same strack
>> > trace and in the code its at the job.getJob().getCounters() part) I
>> > have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any
>> > one had come acrossed this before or had any idea
>> >
>> >
>> >
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> > to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> > reader of
>> > this message is not the intended recipient, you are hereby notified that
>> > any
>> > printing, copying, dissemination, distribution, disclosure or forwarding
>> > of
>> > this communication is strictly prohibited. If you have received this
>> > communication in error, please contact the sender immediately and delete
>> > it
>> > from your system. Thank You.
>
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills

Mime
View raw message