crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: Two strange crunch exceptions
Date Sat, 15 Feb 2014 18:34:58 GMT
Thanks gabriel and josh : Im back on track with your suggestions!


On Sat, Feb 15, 2014 at 12:48 PM, Josh Wills <jwills@cloudera.com> wrote:

>
> Hey Jay,
>
> Replies inlined.
>
> On Sat, Feb 15, 2014 at 9:00 AM, Gabriel Reid <gabriel.reid@gmail.com>wrote:
>
>> Hi Jay,
>>
>> On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <jayunit100@gmail.com> wrote:
>> >
>> > When running MRPipeline here:
>> >
>> https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java
>> >
>> > I got two errors:
>> >
>> > 1) Running the code as-is, I got this very strange stacktrace:
>> >
>> > Exception in thread "Thread-3" java.lang.VerifyError: class
>> > org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method
>> > getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
>>
>
>  You're getting this error b/c you're using an older version of Crunch
> (0.8.2, maybe?) with Hadoop 2.2. You need to use an 0.9.0 version of
> Crunch, which is compiled against the Hadoop 2.2 APIs. The getting started
> guide has a section on "Which Version of Crunch do I need?" that maps
> Hadoop and HBase version IDs to the corresponding Crunch release:
>
> http://crunch.apache.org/getting-started.html
>
>
>> It looks like there's a version issue with protobufs in your project
>> (it seems yarn is dependent on version 2.5.x, and it's version 2.4
>> that is getting pulled in. There are a number of ways to get around
>> this, but easiest one is to add an explicit dependency to the
>> protobuf-java at the top of your pom, i.e.
>>
>>        <dependency>
>>             <groupId>com.google.protobuf</groupId>
>>             <artifactId>protobuf-java</artifactId>
>>             <version>2.5.0</version>
>>         </dependency>
>>
>>
>>
>> > 2)  I changed line 45  to a "MemPipeline", and a new error took its
>> place:
>> >
>> > java.lang.IncompatibleClassChangeError: Found interface
>> > org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was
>> expected
>> >     at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127)
>> >     at
>> >
>> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471)
>> >     at ...
>> >
>> > I assume there is something wrong with my environment or job setup, but
>> ive
>> > logically used most of this code before, and am just refactoring, so am
>> > puzzled why all these funny errors have cropped up.
>> >
>>
>> Your project is currently dependent on hadoop-2 (i.e. yarn), and is
>> using a dependency on Crunch that is specific to hadoop-1. There are
>> crunch builds linked to hadoop-2, and you'll need to use one of these
>> in order to run Crunch together with hadoop-2. Changing your Crunch
>> dependency version to 0.8.2-hadoop2 (or any other version with the
>> -hadoop2 suffix) should resolve this.
>>
>> Hope this helps!
>>
>> - Gabriel
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 
Jay Vyas
http://jayunit100.blogspot.com

Mime
View raw message