hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
Date Fri, 23 Jan 2009 03:18:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666404#action_12666404

Chris Douglas commented on HADOOP-4975:

     [exec] +1 overall.  
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>                 Key: HADOOP-4975
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4975
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Jingkei Ly
>            Assignee: Jingkei Ly
>         Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader
is loaded key and value classes which are required to read records from SequenceFiles that
were written out in a previous MapReduce job. 
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to
create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader
tries to create an instance of the Key/Value classes, presumably because the class loader
SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an
example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> 	at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> 	at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> 	... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> 	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> 	at java.lang.Class.forName0(Native Method)
> 	at java.lang.Class.forName(Class.java:247)
> 	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> 	at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> 	... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message