crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <gabriel.r...@gmail.com>
Subject Re: Two strange crunch exceptions
Date Sat, 15 Feb 2014 17:00:44 GMT
Hi Jay,

On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <jayunit100@gmail.com> wrote:
>
> When running MRPipeline here:
> https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java
>
> I got two errors:
>
> 1) Running the code as-is, I got this very strange stacktrace:
>
> Exception in thread "Thread-3" java.lang.VerifyError: class
> org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method
> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;

It looks like there's a version issue with protobufs in your project
(it seems yarn is dependent on version 2.5.x, and it's version 2.4
that is getting pulled in. There are a number of ways to get around
this, but easiest one is to add an explicit dependency to the
protobuf-java at the top of your pom, i.e.

       <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>2.5.0</version>
        </dependency>



> 2)  I changed line 45  to a "MemPipeline", and a new error took its place:
>
> java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
>     at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127)
>     at
> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471)
>     at ...
>
> I assume there is something wrong with my environment or job setup, but ive
> logically used most of this code before, and am just refactoring, so am
> puzzled why all these funny errors have cropped up.
>

Your project is currently dependent on hadoop-2 (i.e. yarn), and is
using a dependency on Crunch that is specific to hadoop-1. There are
crunch builds linked to hadoop-2, and you'll need to use one of these
in order to run Crunch together with hadoop-2. Changing your Crunch
dependency version to 0.8.2-hadoop2 (or any other version with the
-hadoop2 suffix) should resolve this.

Hope this helps!

- Gabriel

Mime
View raw message