hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Fri, 12 Nov 2010 22:10:19 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931541#action_12931541
] 

Chris Douglas commented on HADOOP-6685:
---------------------------------------

bq. SequenceFile supports non-Writable types already. The limitation today is that there must
be a one-to-one mapping between Java class and serialized data type.

Of course; I misspoke.

bq.  I think that can be satisfied by both Thrift and Protocol Buffers. For Avro, I don't
think we want to support it in SequenceFile, as we should instead encourage use of Avro Data
File

Let's leave it up to the user. Leaving Avro out doesn't make sense, either; I'm sure there
are more efficient formats for protocol buffers, too.

bq. A process question: given how we failed to gain consensus last time, what could we do
differently this time round?

If the JIRA stays simple and on-topic, I think consensus is achievable. If the argument strays
into the order of issues, philosophical musings on the future of MapReduce, and rehashing
past debates, then failure is also achievable. If participants behave as if its success were
more important than "[victory|http://xkcd.com/386/]," I think the purpose is clear enough
to anyone hoping to contribute work. What questions do you want answered in a design document?

> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: serial.patch
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message