crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <>
Subject Re: Two strange crunch exceptions
Date Sat, 15 Feb 2014 17:00:44 GMT
Hi Jay,

On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <> wrote:
> When running MRPipeline here:
> I got two errors:
> 1) Running the code as-is, I got this very strange stacktrace:
> Exception in thread "Thread-3" java.lang.VerifyError: class
> org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method
> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;

It looks like there's a version issue with protobufs in your project
(it seems yarn is dependent on version 2.5.x, and it's version 2.4
that is getting pulled in. There are a number of ways to get around
this, but easiest one is to add an explicit dependency to the
protobuf-java at the top of your pom, i.e.


> 2)  I changed line 45  to a "MemPipeline", and a new error took its place:
> java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
>     at org.apache.crunch.DoFn.getConfiguration(
>     at
> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(
>     at ...
> I assume there is something wrong with my environment or job setup, but ive
> logically used most of this code before, and am just refactoring, so am
> puzzled why all these funny errors have cropped up.

Your project is currently dependent on hadoop-2 (i.e. yarn), and is
using a dependency on Crunch that is specific to hadoop-1. There are
crunch builds linked to hadoop-2, and you'll need to use one of these
in order to run Crunch together with hadoop-2. Changing your Crunch
dependency version to 0.8.2-hadoop2 (or any other version with the
-hadoop2 suffix) should resolve this.

Hope this helps!

- Gabriel

View raw message