avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacob Metcalf (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AVRO-1103) New AvroDeserializer should Locate Appropriate Classloader
Date Sat, 26 May 2012 18:12:22 GMT
Jacob Metcalf created AVRO-1103:

             Summary: New AvroDeserializer should Locate Appropriate Classloader
                 Key: AVRO-1103
                 URL: https://issues.apache.org/jira/browse/AVRO-1103
             Project: Avro
          Issue Type: Improvement
          Components: java
    Affects Versions: 1.7.0
         Environment: Hadoop 0.23.1 with Avro jars replaced by 1.7 jars
Specific data classes assembled into JAR with mapper/reducer
            Reporter: Jacob Metcalf

Continuing on from AVRO-873 I believe some more work needs to be done to get the MapReduce
2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves around classloaders it is complex
to present a unit test which fails so I will explain the problem:

- By default SpecificDatumReader will use the classloader it was loaded from to find a Specific
class to deserialize into.

- In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so typically you would bundle
Avro into your job jar along with the Specific classes so they would be on the same classpath.
- However later versions of Hadoop such as 0.23 ship with Avro. Thus you find that the SpecificData.class.getClassloader()
is typically a parent loader which just contains Hadoop components.

- Thus when SpecificData goes to construct a Specific class from the schema it cannot locate
it and silently defaults to creating a GenericData.

In AVRO-873 an additional constructor was added to SpecificData to force it to use a different
classloader. Thus to extend this fix to the new MR2 APIs:

- AvroDeserializer could attempt to instantiate the class using Class.forName() and from this
get the appropriate Classloader and pass this into the constructor of SpecificDatumReader.

- Line 51 of SpecificData.java is:

bq. protected SpecificData() { this(SpecificData.class.getClassLoader()); }

- This would need to be changed to use this.classloader.

I have raised this in the mail groups here: http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded
so apologies if this is already being thought about.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message