avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <scottca...@apache.org>
Subject Re: Java MapReduce Avro Jackson Error
Date Tue, 20 Mar 2012 03:25:59 GMT
If you are using avro-tools, beware it is a shaded jar with all dependencies
inside of it for use as a command line tool (java ­jar
avro-tools-VERSION.jar).

If you are using avro-tools in your project for some reason (there is really
only command line utilities in it) use the nodeps classifier:

<classifier>nodeps</classifier>

http://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.6.3/

Note the nodeps jar is 47K, while the default jar is 10MB.


For what it is worth, I removed the Jackson jar from our hadoop install long
ago.  It is used to dump configuration files to JSON there, a peripheral
feature we don't use.

Another thing that you may want to do is change your Hadoop dependency scope
to
<scope>provided</scope> since hadoop will be put on your classpath by the
hadoop environment.   Short of this, excluding the chained Hadoop
dependencies you aren't using (most likely: jetty,  kfs, and the
tomcat:jasper and eclipse:jdt stuff) may help.

On 3/19/12 6:23 PM, "Deepak Nettem" <deepaknettem@gmail.com> wrote:

> Hi Tatu / Scott,
> 
> Thanks for your replies. I replaced the earlier dependencies with these:
> 
>    <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro-tools</artifactId>
>     <version>1.6.3</version>
>     </dependency>
>     
>     <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro</artifactId>
>     <version>1.6.3</version>
>     </dependency>
> 
>     <dependency> 
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-mapper-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
> 
>     <dependency> 
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-core-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
> 
> And this is my app dependency tree:
> 
> [INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ AvroTest ---
> [INFO] org.avrotest:AvroTest:jar:1.0-SNAPSHOT
> [INFO] +- junit:junit:jar:3.8.1:test (scope not updated to compile)
> [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
> [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
> [INFO] +- net.sf.json-lib:json-lib:jar:jdk15:2.3:compile
> [INFO] |  +- commons-beanutils:commons-beanutils:jar:1.8.0:compile
> [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] |  +- commons-lang:commons-lang:jar:2.4:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
> [INFO] |  \- net.sf.ezmorph:ezmorph:jar:1.0.6:compile
> [INFO] +- org.apache.avro:avro-tools:jar:1.6.3:compile
> [INFO] |  \- org.slf4j:slf4j-api:jar:1.6.4:compile
> [INFO] +- org.apache.avro:avro:jar:1.6.3:compile
> [INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
> [INFO] \- org.apache.hadoop:hadoop-core:jar:0.20.2:compile
> [INFO]    +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]    +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]    +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
> [INFO]    +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]    +- commons-net:commons-net:jar:1.4.1:compile
> [INFO]    +- org.mortbay.jetty:jetty:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jetty-util:jar:6.1.14:compile
> [INFO]    +- tomcat:jasper-runtime:jar:5.5.12:compile
> [INFO]    +- tomcat:jasper-compiler:jar:5.5.12:compile
> [INFO]    +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
> [INFO]    |  \- ant:ant:jar:1.6.5:compile
> [INFO]    +- commons-el:commons-el:jar:1.0:compile
> [INFO]    +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]    +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
> [INFO]    +- net.sf.kosmosfs:kfs:jar:0.3:compile
> [INFO]    +- hsqldb:hsqldb:jar:1.8.0.10:compile
> [INFO]    +- oro:oro:jar:2.0.8:compile
> [INFO]    \- org.eclipse.jdt:core:jar:3.1.1:compile
> 
> I still get the same error. Is there anything specific I need to do other than
> changing dependencies in pom.xml to make this error go away?
> 
> On Mon, Mar 19, 2012 at 9:12 PM, Tatu Saloranta <tsaloranta@gmail.com> wrote:
>> On Mon, Mar 19, 2012 at 6:06 PM, Scott Carey <scottcarey@apache.org> wrote:
>>> > What version of Avro are you using?
>>> >
>>> > You may want to try Avro 1.6.3 + Jackson 1.8.8.
>>> >
>>> > This is related, but is not your exact problem.
>>> > https://issues.apache.org/jira/browse/AVRO-1037
>>> >
>>> > You are likely pulling in some other version of jackson somehow.  You may
>>> > want to use 'mvn dependency:tree' on your project to see where all the
>>> > dependencies are coming from.  That may help identify the culprit.
>> 
>> This sounds like a good idea, and I agree in that this is probably
>> still due to an old version lurking somewhere.
>> 
>> -+ Tatu +-
> 
> 



Mime
View raw message