hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Mon, 13 Dec 2010 18:46:10 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970947#action_12970947
] 

Doug Cutting commented on HADOOP-6685:
--------------------------------------

> But your real concern is that the configuration key is also typed binary [ ... ]

My concern is that in the proposal here there are no configuration keys, textual, binary or
otherwise. As configurations (like serializatons, codecs, etc.) are nested inside other configurations
(like inputformats, mappers, etc.) they become hidden.  Values may only be specified through
Java APIs.  One might set the reducer's compression level in a Java program with something
like the following:

 ((GzipCodec)((SequenceFileOutputFormat)myReducer.getOutputFormat()).getCodec()).setLevel(9)

But it would be nice if one could also specify this with something like -Dreducer.outputformat.codec.level=9.
 A common configuration data model would make this easy to implement, as it is today.  Strongly
typed values, nested configurations and binary values would all be useful things to add to
this data model.  But discarding attribute-based access in the process seems like a mistake.

> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: serial.patch, serial4.patch, serial6.patch, serial7.patch, serial9.patch,
SerializationAtSummit.pdf
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message