spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gil Vernik <G...@il.ibm.com>
Subject run time exceptions in Spark 1.2.0 manual build together with OpenStack hadoop driver
Date Sun, 18 Jan 2015 06:26:38 GMT
Hi,

I took a source code of Spark 1.2.0 and tried to build it together with 
hadoop-openstack.jar ( To allow Spark an access to OpenStack Swift )
I used Hadoop 2.6.0.

The build was fine without problems, however in run time, while trying to 
access "swift://" name space i got an exception:
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
                 at 
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
                 at 
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
                ...and the long stack trace goes here

Digging into the problem i saw the following:
Jackson versions 1.9.X are not backward compatible, in particular they 
removed JsonClass annotation.
Hadoop 2.6.0 uses jackson-asl version 1.9.13, while Spark has reference to 
older version of jackson.

This is the main  pom.xml of Spark 1.2.0 :

      <dependency>
        <!-- Matches the version of jackson-core-asl pulled in by avro -->
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.8.8</version>
      </dependency>

Referencing 1.8.8 version, which is not compatible with Hadoop 2.6.0 .
If we change version to 1.9.13, than all will work fine and there will be 
no run time exceptions while accessing Swift. The following change will 
solve the problem:

      <dependency>
        <!-- Matches the version of jackson-core-asl pulled in by avro -->
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.13</version>
      </dependency>

I am trying to resolve this somehow so people will not get into this 
issue.
Is there any particular need in Spark for jackson 1.8.8 and not 1.9.13?
Can we remove 1.8.8 and put 1.9.13 for Avro? 
It looks to me that all works fine when Spark build with jackson 1.9.13, 
but i am not an expert and not sure what should be tested.

Thanks,
Gil Vernik.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message