hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Tue, 16 Nov 2010 20:34:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932644#action_12932644
] 

Owen O'Malley commented on HADOOP-6685:
---------------------------------------

@Luke - I agree it is a last resort. They haven't been able to figure out how to push to Maven.
I asked and they aren't sure what is going wrong. I spent a couple hours trying to see if
I could get it pushed to org.apache.hadoop.thrift and didn't succeed. 

@Tom - But they aren't optional dependences. You could make the same argument for log4j, commons
logging or any other package we use. Part of the power of Maven is that it catches these problems
early and gives you tools for resolving them. Making them optional would leave the users in
the lurch.

Serializations have complete control over how they serialize their metadata. Any or all of
them could use json or any other format. Using protobuf is very convenient and provides lots
of extensibility. I believe that using a non-extensible format would be a mistake we'd regret
for a long time.

 



> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: libthrift.jar, serial.patch, serial4.patch, serial6.patch, SerializationAtSummit.pdf
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message