hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Parks (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
Date Tue, 19 Mar 2013 12:15:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606271#comment-13606271

David Parks commented on HADOOP-9295:

Your test missed the problem. You added two Text objects, you need to add 2 custom MapWrtiable
objects to the map to trigger the problem.

The reason explained:

Notice this code in AbstractMapWritable:

	    addToMap(ArrayWritable.class, Byte.valueOf(Integer.valueOf(-127).byteValue())); 
	    addToMap(BooleanWritable.class, Byte.valueOf(Integer.valueOf(-126).byteValue()));
	    addToMap(BytesWritable.class, Byte.valueOf(Integer.valueOf(-125).byteValue()));
	    addToMap(FloatWritable.class, Byte.valueOf(Integer.valueOf(-124).byteValue()));
	    addToMap(IntWritable.class, Byte.valueOf(Integer.valueOf(-123).byteValue()));
	    addToMap(LongWritable.class, Byte.valueOf(Integer.valueOf(-122).byteValue()));
	    addToMap(MapWritable.class, Byte.valueOf(Integer.valueOf(-121).byteValue()));
	    addToMap(MD5Hash.class, Byte.valueOf(Integer.valueOf(-120).byteValue()));
	    addToMap(NullWritable.class, Byte.valueOf(Integer.valueOf(-119).byteValue()));
	    addToMap(ObjectWritable.class, Byte.valueOf(Integer.valueOf(-118).byteValue()));
	    addToMap(SortedMapWritable.class, Byte.valueOf(Integer.valueOf(-117).byteValue()));
	    addToMap(Text.class, Byte.valueOf(Integer.valueOf(-116).byteValue()));
	    addToMap(TwoDArrayWritable.class, Byte.valueOf(Integer.valueOf(-115).byteValue()));
	    // UTF8 is deprecated so we don't support it
	    addToMap(VIntWritable.class, Byte.valueOf(Integer.valueOf(-114).byteValue()));
	    addToMap(VLongWritable.class, Byte.valueOf(Integer.valueOf(-113).byteValue()));

It's adding the "typical" Writables to the class map by default, so any of these classes always
maps correctly, this is probably why the problem was never noticed before now.

It's only when you add a Writable object that isn't already in this list that it has to add
it to the map, and thus encounters the bug.
> AbstractMapWritable throws exception when calling readFields() multiple times when the
maps contain different class types
> -------------------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-9295
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9295
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 1.0.3
>            Reporter: David Parks
>            Assignee: Karthik Kambatla
>            Priority: Critical
>         Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
> Verified the trunk looks the same as 1.0.3 for this issue.
> When mappers output MapWritables with different class types, then they are read in on
the Reducer via an iterator (multiple calls to readFields without instantiating a new object)
you'll get this:
> java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not
>         at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
>         at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
> It happens because AbstractMapWritable accumulates class type entries in its ClassType
to ID (and vice versa) hashmaps.
> Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls
to readFields().
> I've attached a JUnit test that both demonstrates the problem and contains an embedded,
fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where
it was fixed in 2 places).
> If there's a better way to submit this recommended bug fix, someone please feel free
to let me know.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message