hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Thu, 02 Dec 2010 19:54:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966250#action_12966250
] 

Doug Cutting commented on HADOOP-6685:
--------------------------------------

> I guess I could add a method to Serialization that parses the input from a String containing
a serialization-specified format that was a user-friendly DSL.

Yes, that would be useful.

Since configurations are meant to be nestable, a given string format would need to be able
to nest other, different string formats.  So each string format would probably need to escape
strings accordingly.  The need for escaping could be avoided if instead each serialization
had a toJsonString and fromJsonString method, since nesting JSON within JSON is easy.

Also, once either toString/fromString or toJsonString/fromJsonString methods are added to
Serialization, then there's no need for each serialization to also implement serializeSelf
and deserializeSelf, since these can simply become convenience methods implemented in terms
of the String-based methods.  For better performance, if that's a concern for configuration
data, the core methods might instead be toString(Writer) and fromString(Reader).


> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: libthrift.jar, serial.patch, serial4.patch, serial6.patch, serial7.patch,
SerializationAtSummit.pdf
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message