crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Sparks <jspa...@cray.com>
Subject Re: Yarnchild error : crunch-0.7.0
Date Tue, 25 Feb 2014 21:54:25 GMT
So interesting … same results.

This time I ran two versions 1) the examples from the crunch build and the other 2) a standalone
application. The result for the standalone was the same as before  - I guess I expected that.
The other failure was different and a little more confusing. I guess the question I have is
can this be caused by the JDK used to build crunch. We are using JDK1.7

Failure 1)

2014-02-25 14:59:04,252 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : org.apache.crunch.CrunchRuntimeException: Could not read runtime node information
at org.apache.crunch.impl.mr.run.CrunchTaskContext.<init>(CrunchTaskContext.java:48)
at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(AccessController.java:366)
at javax.security.auth.Subject.doAs(Subject.java:572)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.io.InvalidClassException: org.apache.crunch.types.writable.Writables$4; local
class incompatible: stream classdesc serialVersionUID = 5855040850180329703, local class serialVersionUID
= 4130080921736307351

Failure 2)
2014-02-25 14:59:33,926 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child
: java.lang.IncompatibleClassChangeError: org/apache/hadoop/mapreduce/JobContext.getConfiguration()Lorg/apache/hadoop/conf/Configuration;
at org.apache.crunch.impl.mr.run.CrunchTaskContext.<init>(CrunchTaskContext.java:42)
at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(AccessController.java:366)
at javax.security.auth.Subject.doAs(Subject.java:572)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)


JDK
jsparks@jupiter:/lus/dal/jsparks/example/tmp/hdlogs.jsparks/userlogs> java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.

From: Josh Wills <josh.wills@gmail.com<mailto:josh.wills@gmail.com>>
Reply-To: "user@crunch.apache.org<mailto:user@crunch.apache.org>" <user@crunch.apache.org<mailto:user@crunch.apache.org>>
Date: Tuesday, February 25, 2014 1:54 PM
To: "user@crunch.apache.org<mailto:user@crunch.apache.org>" <user@crunch.apache.org<mailto:user@crunch.apache.org>>
Subject: Re: Yarnchild error : crunch-0.7.0

Yeah, try it again w/ -Dcrunch.platform=2 instead of -Dhadoop.profile=2.0

J


On Tue, Feb 25, 2014 at 11:47 AM, Bill Sparks <jsparks@cray.com<mailto:jsparks@cray.com>>
wrote:
Well I did the following and also changed the pom.xml to reference the correct hadoop version.

$ mvn clean install -Dhadoop.profile=2.0 –DskipTests

<hadoop.version>2.0.6-alpha</hadoop.version>

--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.

From: Josh Wills <jwills@cloudera.com<mailto:jwills@cloudera.com>>
Reply-To: "user@crunch.apache.org<mailto:user@crunch.apache.org>" <user@crunch.apache.org<mailto:user@crunch.apache.org>>
Date: Tuesday, February 25, 2014 1:43 PM
To: "user@crunch.apache.org<mailto:user@crunch.apache.org>" <user@crunch.apache.org<mailto:user@crunch.apache.org>>
Subject: Re: Yarnchild error : crunch-0.7.0

Hrm-- that's usually related to the API changes between hadoop1 and hadoop2. How did you build
crunch, exactly? Did you use -Dcrunch.platform=2?

J


On Tue, Feb 25, 2014 at 11:37 AM, Bill Sparks <jsparks@cray.com<mailto:jsparks@cray.com>>
wrote:
Can anyone shed some light on why I would be getting the following error when submitting a
simple crunch wordcount example. Other Hadoop MR applications work, just it seems that Crunch
is confused about some class definitions.

I'm running hadoop-2.0.6-alpha and have build crunch to match.

Hadoop 2.0.6-alpha
Subversion Unknown -r ca4c88898f95aaab3fd85b5e9c194ffd647c2109
Compiled by jenkins on 2013-10-30T07:19Z
>From source with checksum 95e88b2a9589fa69d6d5c1dbd48d4e


2014-02-25 13:23:00,049 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid
= 268435456
2014-02-25 13:23:00,049 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 67108860;
length = 16777216
2014-02-25 13:23:00,070 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child
: java.lang.IncompatibleClassChangeError: org/apache/hadoop/mapreduce/JobContext.getConfiguration()Lorg/apache/hadoop/conf/Configuration;
at org.apache.crunch.impl.mr.run.CrunchTaskContext.<init>(CrunchTaskContext.java:42)
at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(AccessController.java:366)
at javax.security.auth.Subject.doAs(Subject.java:572)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)




--
Director of Data Science
Cloudera<http://www.cloudera.com>
Twitter: @josh_wills<http://twitter.com/josh_wills>


Mime
View raw message