hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Fri, 19 Oct 2007 10:40:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536176

Vivek Ratan commented on HADOOP-1986:

>> Another aspect that the current patch doesn't address is who instantiates objects
during deserialization. 

Good point. Maybe this is what Doug and/or you mean, but you could have something like this:
boolean acceptObjectReference(); // returns true if the framework deserializes into an object,
false if it creates an object and returns it
Object deserialize(Object reuse, InputStream);

Frameworks that expect the caller to create an object before deserializing it (Thrift, Record
I/O), would return NULL, but others that create their own objects would accept a NULL value
for the 'reuse' parameter. The _acceptObjectReference()_ method tells a client which option
the deserializer prefers, though most clients would already know before-hand. Is this similar
to what you were thinking about? 

Another option is to have separate deserialize methods: 
Object deserialize(InputStream);
void deserialize(Object, InputStream);
Most frameworks would implement only one of these methods, perhaps throwing an exception for
the one they don't implement (you'd also need a way for a client to dynamically find out which
frameworks implements which method).

I lean towards the former - it's more compact, though the latter is a little more cleaner
(less confusion). 

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>         Attachments: SerializableWritable.java, serializer-v1.patch
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message