hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Mon, 15 Oct 2007 05:20:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534738

Vivek Ratan commented on HADOOP-1986:

After some offline discussions, I think there is some convergence. It seems like most folks
have implicitly assumed that DDLs are involved. If a user defines a class that they want to
use as a key or value, they (or somebody) would either implement Writable or define it using
a DDL, run a Thrift or Record I/O compiler, and use the generated stub. They would also pick
the right _Serializable<T>_ implementation. I've been arguing that there is an additional
way where someone may not want to go through the pain of writing a DDL and compiling it. I
don't have any real use cases for assuming that some folks might find DDLs and compilers to
be a pain. A fair compromise is to stick with the _Serializable<T>_ interface that Tom
and Owen have been talking about. There will be classes that implement _Serializable<RecordI/O
record>_ and _Serializable<Thrift record>_. There can be an additional implementation
for a general-purpose reflection-based serializer, something like _class ReflectionSerializer
imlements Serializable<Object>_. I don't know if such an implementation is required
right away: it's useful only if someone doesn't want to deal with DDL hassles. So maybe we
can keep this as an option for later, if required. 

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>         Attachments: SerializableWritable.java
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message