hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Thu, 18 Nov 2010 23:59:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933615#action_12933615
] 

Eli Collins commented on HADOOP-6685:
-------------------------------------

It seems like the high-level discussion got side tracked on the use of specific serialization
frameworks, instead of getting consensus on the main change this jira proposes: whether the
framework should use an opaque binary blob for storing metadata, ie whether serialization
metadata needs to be serialized (vs a map of strings which is, which is really just a different
form of serialization). 

Reading the thread and presentation it sounds like the primary motivation is:
# Using a binary blob will simplify the interfaces for defining specific serializations (MR-1462)

# The blob interface allows for contents that are more easily type checked

If there is consensus on these points?  If not, what are the specific objections? Eg is the
patch for MR-1462 not significantly simplified vs what it would take with the current API?


It doesn't seem to me that this particular plumbing decision needs to block the more valuable
goal of pushing serializations through the pipeline.

> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: libthrift.jar, serial.patch, serial4.patch, serial6.patch, SerializationAtSummit.pdf
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message