hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Wed, 28 Nov 2007 21:58:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546424
] 

Tom White commented on HADOOP-1986:
-----------------------------------

I've attached a second patch, serializer-v2.patch, which has the Serialization abstraction
and the object reuse facility for Deserializers that Doug suggested. Tests, warnings and javadoc
still need attending to.

I've also included some examples that show how to use different serialization frameworks for
running MapReduce programs. It includes Writable, Thrift, Record IO and Java Serialization
examples. It might be good to improve these so they can be included in Hadoop - they are pretty
rough and ready at the moment, which is why they are a standalone tarball.

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>
>         Attachments: hadoop-serializer-v2.tar.gz, SerializableWritable.java, serializer-v1.patch,
serializer-v2.patch
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message