crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: Yarnchild error : crunch-0.7.0
Date Tue, 25 Feb 2014 19:54:56 GMT
Yeah, try it again w/ -Dcrunch.platform=2 instead of -Dhadoop.profile=2.0

J


On Tue, Feb 25, 2014 at 11:47 AM, Bill Sparks <jsparks@cray.com> wrote:

>  Well I did the following and also changed the pom.xml to reference the
> correct hadoop version.
>
>  $ mvn clean install -Dhadoop.profile=2.0 -DskipTests
>
>  <hadoop.version>2.0.6-alpha</hadoop.version>
>
>  --
>  Jonathan (Bill) Sparks
> Software Architecture
> Cray Inc.
>
>   From: Josh Wills <jwills@cloudera.com>
> Reply-To: "user@crunch.apache.org" <user@crunch.apache.org>
> Date: Tuesday, February 25, 2014 1:43 PM
> To: "user@crunch.apache.org" <user@crunch.apache.org>
> Subject: Re: Yarnchild error : crunch-0.7.0
>
>   Hrm-- that's usually related to the API changes between hadoop1 and
> hadoop2. How did you build crunch, exactly? Did you use
> -Dcrunch.platform=2?
>
>  J
>
>
> On Tue, Feb 25, 2014 at 11:37 AM, Bill Sparks <jsparks@cray.com> wrote:
>
>>  Can anyone shed some light on why I would be getting the following
>> error when submitting a simple crunch wordcount example. Other Hadoop MR
>> applications work, just it seems that Crunch is confused about some class
>> definitions.
>>
>>  I'm running hadoop-2.0.6-alpha and have build crunch to match.
>>
>>  Hadoop 2.0.6-alpha
>> Subversion Unknown -r ca4c88898f95aaab3fd85b5e9c194ffd647c2109
>> Compiled by jenkins on 2013-10-30T07:19Z
>> From source with checksum 95e88b2a9589fa69d6d5c1dbd48d4e
>>
>>
>>  2014-02-25 13:23:00,049 INFO [main] org.apache.hadoop.mapred.MapTask:
>> bufstart = 0; bufvoid = 268435456
>> 2014-02-25 13:23:00,049 INFO [main] org.apache.hadoop.mapred.MapTask:
>> kvstart = 67108860; length = 16777216
>> 2014-02-25 13:23:00,070 FATAL [main] org.apache.hadoop.mapred.YarnChild:
>> Error running child : java.lang.IncompatibleClassChangeError:
>> org/apache/hadoop/mapreduce/JobContext.getConfiguration()Lorg/apache/hadoop/conf/Configuration;
>> at
>> org.apache.crunch.impl.mr.run.CrunchTaskContext.<init>(CrunchTaskContext.java:42)
>> at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:37)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
>> at java.security.AccessController.doPrivileged(AccessController.java:366)
>> at javax.security.auth.Subject.doAs(Subject.java:572)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
>>
>>
>
>
>  --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Mime
View raw message