crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Two strange crunch exceptions
Date Sat, 15 Feb 2014 17:48:43 GMT
Hey Jay,

Replies inlined.

On Sat, Feb 15, 2014 at 9:00 AM, Gabriel Reid <gabriel.reid@gmail.com>wrote:

> Hi Jay,
>
> On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <jayunit100@gmail.com> wrote:
> >
> > When running MRPipeline here:
> >
> https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java
> >
> > I got two errors:
> >
> > 1) Running the code as-is, I got this very strange stacktrace:
> >
> > Exception in thread "Thread-3" java.lang.VerifyError: class
> > org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method
> > getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
>

 You're getting this error b/c you're using an older version of Crunch
(0.8.2, maybe?) with Hadoop 2.2. You need to use an 0.9.0 version of
Crunch, which is compiled against the Hadoop 2.2 APIs. The getting started
guide has a section on "Which Version of Crunch do I need?" that maps
Hadoop and HBase version IDs to the corresponding Crunch release:

http://crunch.apache.org/getting-started.html


> It looks like there's a version issue with protobufs in your project
> (it seems yarn is dependent on version 2.5.x, and it's version 2.4
> that is getting pulled in. There are a number of ways to get around
> this, but easiest one is to add an explicit dependency to the
> protobuf-java at the top of your pom, i.e.
>
>        <dependency>
>             <groupId>com.google.protobuf</groupId>
>             <artifactId>protobuf-java</artifactId>
>             <version>2.5.0</version>
>         </dependency>
>
>
>
> > 2)  I changed line 45  to a "MemPipeline", and a new error took its
> place:
> >
> > java.lang.IncompatibleClassChangeError: Found interface
> > org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was
> expected
> >     at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127)
> >     at
> >
> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471)
> >     at ...
> >
> > I assume there is something wrong with my environment or job setup, but
> ive
> > logically used most of this code before, and am just refactoring, so am
> > puzzled why all these funny errors have cropped up.
> >
>
> Your project is currently dependent on hadoop-2 (i.e. yarn), and is
> using a dependency on Crunch that is specific to hadoop-1. There are
> crunch builds linked to hadoop-2, and you'll need to use one of these
> in order to run Crunch together with hadoop-2. Changing your Crunch
> dependency version to 0.8.2-hadoop2 (or any other version with the
> -hadoop2 suffix) should resolve this.
>
> Hope this helps!
>
> - Gabriel
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message