avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: java.lang.NoSuchMethodError: org.codehaus.jackson.JsonFactory.enable
Date Mon, 24 Jan 2011 23:13:13 GMT
That is confusing.  Can you capture the classpath of an actual task process, not just the TT?
 They shouldn't differ much, but it is worth checking.

Jackson 1.3 (or was it 1.2?) and above have all been backwards compatible with each other
I believe.   And the error you are getting is definitely caused by accessing the enable()
methods that were added after 1.0.1.
I can change the Avro dependency on Jackson to 1.5.5, 1.7.1, or 1.3, and unit tests pass.
 If I change it to 1.2, 1.1, or 1.0.1 they break.

-Scott

On 1/24/11 2:40 PM, "felix gao" <gre1600@gmail.com<mailto:gre1600@gmail.com>>
wrote:

sorry for the spamming, but I forgot to include the TaskTracker's process information.

root     20368     1  0 17:31 pts/0    00:00:00 su mapred -s /usr/java/default/bin/java --
-Dproc_tasktracker -Xmx1000m -Dhadoop.log.dir=/usr/lib/hadoop-0.20/logs -Dhadoop.log.file=hadoop-hadoop-tasktracker-ip-10-212-86-214.log
-Dhadoop.home.dir=/usr/lib/hadoop-0.20 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA
-Djava.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
-Dhadoop.policy.file=hadoop-policy.xml -classpath /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.6.jar:/usr/lib/hadoop-0.20/lib/hadoop-thriftfs-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.5.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.5.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo.0.4.4.jar
org.apache.hadoop.mapred.TaskTracker

as you can see the correct version of jackson library is in the classpath.


On Mon, Jan 24, 2011 at 2:09 PM, felix gao <gre1600@gmail.com<mailto:gre1600@gmail.com>>
wrote:
Chase,

I tried to ran it on my local box with a standalone version of hadoop installed and I still
got the same error.
/usr/local/pig-0.7.0/bin/pig avro.pig
11/01/24 14:01:29 INFO pig.Main: Logging error messages to: /Users/felix/Documents/pig/pig_1295906489023.log
2011-01-24 14:01:29,328 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
- Connecting to hadoop file system at: file:///
2011-01-24 14:01:30,277 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing
JVM Metrics with processName=JobTracker, sessionId=
2011-01-24 14:01:30,537 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled
internal error. org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;

Any ideas why?

Thanks,

Felix




On Mon, Jan 24, 2011 at 1:18 PM, Chase Bradford <chase.bradford@gmail.com<mailto:chase.bradford@gmail.com>>
wrote:
Yes, you will need to restart them.  The child tasks inherit the TT's
classpath, which will list only the 1.0.1 jars until you restart the
daemon.

On Mon, Jan 24, 2011 at 1:11 PM, felix gao <gre1600@gmail.com<mailto:gre1600@gmail.com>>
wrote:
> Hi Guys,
>
> I am testing out AVRO in our cluster and having java.lang.NoSuchMethodError:
> org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
> when running a simple pig script.
> after took a look at AVRO-493 I  uploaded the new jackson jar to replace the
> CDH2's jackson 1.0.1 jars.
>
>
> The pigscipt looks like below
> REGISTER /home/pig/jars/avro-1.4.1.jar
> REGISTER /home/pig/jars/json_simple-1.1.jar
> REGISTER /home/pig/jars/piggybank.jar
> REGISTER /usr/lib/hadoop/lib/jackson-core-asl-1.5.5.jar
> REGISTER /usr/lib/hadoop/lib/jackson-mapper-asl-1.5.5.jar
>
> log_load = LOAD '/user/felix/avro_input/*.avro' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage() ;
>
> dump log_load
>
>
> jackson jars are copied to each slaves and master
>
> The full stacktrace:
> ERROR 2998: Unhandled internal error.
> org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
>
> java.lang.NoSuchMethodError:
> org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
>         at org.apache.avro.Schema.<clinit>(Schema.java:82)
>         at
> org.apache.pig.piggybank.storage.avro.ASCommons.<clinit>(ASCommons.java:44)
>         at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:177)
>         at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:133)
>         at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:108)
>         at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:233)
>         at
> org.apache.pig.impl.logicalLayer.LOLoad.determineSchema(LOLoad.java:169)
>         at
> org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:150)
>         at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:843)
>         at
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>         at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1164)
>         at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1114)
>         at org.apache.pig.PigServer.registerQuery(PigServer.java:425)
>         at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:737)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
>         at org.apache.pig.Main.main(Main.java:357)
>
> I am wondering if I need to restart tasktracker and jobtracker in order for
> the jackson jars to be picked up?
>
>
>
>
>
>
>
>



--
Chase Bradford


“If in physics there's something you don't understand, you can always
hide behind the uncharted depths of nature. But if your program
doesn't work, there is no obstinate nature. If it doesn't work, you've
messed up.”

- Edsger Dijkstra



Mime
View raw message