hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cameron Pope (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-120) Reading an ArrayWriter does not work because valueClass does not get initialized
Date Fri, 07 Sep 2007 19:43:31 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cameron Pope updated HADOOP-120:
--------------------------------

    Affects Version/s:     (was: 0.12.3)
                       0.15.0
               Status: Patch Available  (was: Open)

I ran into this issue this morning. I'm new to Hadoop so forgive me if I failed to understand
the big picture in submitting this patch.

To address, I added some documentation to ArrayWritable indicating that it needs to be subclassed
to be used as input to Reducers. I also added a check for valueClass being undefined when
calling readFields() - as a relatively new Hadoop user, it took me a while to figure out what
the NPE I got in ReflectionUtils was and what to do with it. 

It looks like some DFS datastructures depend on ArrayWritable being the way that it is, so
it seems that changing the serialization format would break stuff. It might make sense to
create a different ArrayWritable that serialized type information for Hadoop users specifically
for MapReduce applications, but this class should probably document its contract with the
outside world and throw a specific exception when it's violated. I've also added a unit test
to verify the exceptions thrown or not thrown are as expected.

> Reading an ArrayWriter does not work because valueClass does not get initialized
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-120
>             Project: Hadoop
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.15.0
>         Environment: Red Hat  
>            Reporter: Dick King
>            Assignee: Owen O'Malley
>         Attachments: hadoop-120-fix.patch, hadoop-120-workaround.patch
>
>
> If you have a Reducer whose value type is an ArrayWriter it gets enstreamed alright but
at reconstruction type when ArrayWriter::readFields(DataInput in) runs on a DataInput that
has a nonempty ArrayWriter , newInstance fails trying to instantiate the null class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message