hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Wed, 03 Oct 2007 20:35:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532243

Tom White commented on HADOOP-1986:

> Do you have a proposal for how to implement this?

If we follow Owen's suggestion then we can construct a map of types to Serializer classes.
Then, when running 
MapTask or ReduceTask we can use the map to instantiate an appropriate Serializer for each
of the key and the value types.

> In particular, how can we handle something like Thrift, whose instances don't all implement
some interface?

The target class would have to be Object. However, for this to work we would need to have
some notion of precedence so more specific subtypes (like Writable) match first. Also, this
wouldn't allow you to use two different serialization frameworks whose instances only have
a common type of Object. I'm not sure how much of a problem this would be in practice though.

(I just had a look at a Thrift class, generated with release 20070917, and it is tagged with
java.io.Serializable. It would be more useful though it if implemented an interface that defined
the read/write fields.)

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>             Fix For: 0.16.0
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message