hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1727) Make ...hbase.io.MapWritable more generic so that it can be included in ...hadoop.io
Date Tue, 21 Aug 2007 21:40:30 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12521602

Andrzej Bialecki  commented on HADOOP-1727:

I spotted another potential problem, which indirectly comes from the removal of static methods
and fields (which I agree with).

The addToMap method should allow one to explicitly set the id, and the initialization of the
maps in the AbstractMapWritable constructor should use pre-defined id-s (they could start
from -127 for that matter, while automatically added mappings would start from 0 up to 127
- then we could avoid using getKnownClasses(), because all negative id-s would be pre-defined
by definition).

The reason for this proposed change is the future backward compatibility, and extensibility.

First, the compatibility. Let's assume that one day we deprecate ObjectWritable in favor of
GenericWritable, and then in the next release we remove ObjectWritable completely ... oops,
we can no longer read old data, even if we didn't use ObjectWritable at all, because some
of the remaining class->id mappings are now shifted by one. Therefore, the mappings should
be fixed for all remaining classes across releases, and new classes should not re-use old
id-s for a long time.

I propose the following change then:

protected addToMap(Class clazz, byte id);

protected addToMap(Class clazz);

The second method would automatically assign id-s from 0 up to 127 for arbitrary user types,
the first method would be used just to register pre-defined id-s.

Second, the extensibility. In Nutch we will want to subclass MapWritable, and register additional
frequently used types, so that they use pre-defined id-s instead of the serialized mapping,
for compactness. With the current implementation it's not possible to do this.

> Make ...hbase.io.MapWritable more generic so that it can be included in ...hadoop.io
> ------------------------------------------------------------------------------------
>                 Key: HADOOP-1727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1727
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.15.0
>            Reporter: Jim Kellerman
>            Assignee: Jim Kellerman
>            Priority: Minor
>             Fix For: 0.15.0
>         Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
> The class org.apache.hadoop.hbase.io.MapWritable could be made more generic through the
use of ReflectionUtils so that it could support more Map key and value classes. Currently
it supports Map<WritableComparable, Writable> only.
> When more generalized, submit for consideration to be included in org.apache.hadoop.io

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message