hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Mon, 15 Oct 2007 05:20:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534738
] 

Vivek Ratan commented on HADOOP-1986:
-------------------------------------

After some offline discussions, I think there is some convergence. It seems like most folks
have implicitly assumed that DDLs are involved. If a user defines a class that they want to
use as a key or value, they (or somebody) would either implement Writable or define it using
a DDL, run a Thrift or Record I/O compiler, and use the generated stub. They would also pick
the right _Serializable<T>_ implementation. I've been arguing that there is an additional
way where someone may not want to go through the pain of writing a DDL and compiling it. I
don't have any real use cases for assuming that some folks might find DDLs and compilers to
be a pain. A fair compromise is to stick with the _Serializable<T>_ interface that Tom
and Owen have been talking about. There will be classes that implement _Serializable<RecordI/O
record>_ and _Serializable<Thrift record>_. There can be an additional implementation
for a general-purpose reflection-based serializer, something like _class ReflectionSerializer
imlements Serializable<Object>_. I don't know if such an implementation is required
right away: it's useful only if someone doesn't want to deal with DDL hassles. So maybe we
can keep this as an option for later, if required. 

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>
>         Attachments: SerializableWritable.java
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message