hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Fri, 19 Oct 2007 16:28:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536254

Doug Cutting commented on HADOOP-1986:

I don't see the need for 'acceptObjectReference()'.  Clients which wish to reuse objects can,
the first time, pass null.  For subsequent calls they can pass the value returned from prior
call.  This will work well with implementations which reuse and with implementations that
don't, no?

Note also that we don't need an explicit 'accept()' method: if factory.getSerializer(c) returns
null, then that factory does not know how to serialize instances of the class.

Finally, if we have stateful serializers, then we have to make it clear that they must not
do any buffering.  An application should be able to call 'serializer.setOutput(out)', serialize
an instance or two, do other, non-serializer output to 'out', and then serialize more instances,
without calling 'setOutput' again, right?  In the case of Writable, setOutput would just set
the protected 'out' field of a DataOutputStream, and this would all work fine.  That could
be instead done on each call to a 'serialize(Object, OutputStream)' method, but perhaps its
better to factor it out of inner loops.  Is that the intent?

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>         Attachments: SerializableWritable.java, serializer-v1.patch
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message