hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Tue, 02 Oct 2007 22:46:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531938

Owen O'Malley commented on HADOOP-1986:

First, I'd argue that Thrift should have a top level serialization interface... but clearly
that belongs on their development list. *smile*

But if the serializer is specific to the job, you wouldn't be able to mix Writables and Thrift
objects. If you wanted to translate, for instance, you'd like:

map input: MyWritableKey, MyWritableValue
map output: MyThriftKey, MyThriftValue

If I have to have a single serializer for my job, that is a pain. Of course, without a Thrift
record super class, you really can't write ThriftSerializable anyways. (You need a standard
interface to generate the bytes...)

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>             Fix For: 0.16.0
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message