hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Wed, 17 Oct 2007 21:18:50 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated HADOOP-1986:

    Attachment: serializer-v1.patch

Here's a patch (serializer-v1.patch) for discussion. It's still quite incomplete: there are
no tests or javadoc, it produces lots of warnings and I haven't looked at changing pipes or
aggregate to not depend on Writable.

That said, I've managed to run a MapReduce job that doesn't use Writables. (I created a wrapper
round Writable that didn't actually implement Writable, even though the underlying serialization
mechanism used Writable.) The next test is probably to try using Thrift types.


> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>         Attachments: SerializableWritable.java, serializer-v1.patch
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message