hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Tue, 02 Oct 2007 22:46:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531938
] 

Owen O'Malley commented on HADOOP-1986:
---------------------------------------

First, I'd argue that Thrift should have a top level serialization interface... but clearly
that belongs on their development list. *smile*

But if the serializer is specific to the job, you wouldn't be able to mix Writables and Thrift
objects. If you wanted to translate, for instance, you'd like:

map input: MyWritableKey, MyWritableValue
map output: MyThriftKey, MyThriftValue

If I have to have a single serializer for my job, that is a pain. Of course, without a Thrift
record super class, you really can't write ThriftSerializable anyways. (You need a standard
interface to generate the bytes...)

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>             Fix For: 0.16.0
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message