hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Fri, 19 Oct 2007 10:40:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536176
] 

Vivek Ratan commented on HADOOP-1986:
-------------------------------------

>> Another aspect that the current patch doesn't address is who instantiates objects
during deserialization. 

Good point. Maybe this is what Doug and/or you mean, but you could have something like this:
 
{code}
boolean acceptObjectReference(); // returns true if the framework deserializes into an object,
false if it creates an object and returns it
Object deserialize(Object reuse, InputStream);
{code}

Frameworks that expect the caller to create an object before deserializing it (Thrift, Record
I/O), would return NULL, but others that create their own objects would accept a NULL value
for the 'reuse' parameter. The _acceptObjectReference()_ method tells a client which option
the deserializer prefers, though most clients would already know before-hand. Is this similar
to what you were thinking about? 

Another option is to have separate deserialize methods: 
{code}
Object deserialize(InputStream);
void deserialize(Object, InputStream);
{code}
Most frameworks would implement only one of these methods, perhaps throwing an exception for
the one they don't implement (you'd also need a way for a client to dynamically find out which
frameworks implements which method).

I lean towards the former - it's more compact, though the latter is a little more cleaner
(less confusion). 


> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>
>         Attachments: SerializableWritable.java, serializer-v1.patch
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message