hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6685) Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
Date Fri, 19 Nov 2010 20:06:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933950#action_12933950
] 

Arun C Murthy commented on HADOOP-6685:
---------------------------------------

{quote}
> By making the framework use typed metadata instead of the very generic, but type-less,
string to string map many user errors will be avoided.

The current style is to provide methods to access configurations and metadata. These methods
prevent such type errors. I have not seen a large number of complaints from end users about
this aspect of Hadoop.
{quote}

Again, this is exactly the stuff we are trying to improve!

Pardon my frustration, but if you carefully read MAPREDUCE-1183, Configuration just isn't
working! 

Pig (and probably Hive too, I don't know Hive well enough) has gone through significant hoops
to work around it's limitations.

Adding more 'features' on Configuration will exacerbate the issues we have, need to move away
and this is a start.

> Change the generic serialization framework API to use serialization-specific bytes instead
of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: libthrift.jar, serial.patch, serial4.patch, serial6.patch, serial7.patch,
SerializationAtSummit.pdf
>
>
> Currently, the generic serialization framework uses Map<String,String> for the
serialization specific configuration. Since this data is really internal to the specific serialization,
I think we should change it to be an opaque binary blob. This will simplify the interface
for defining specific serializations for different contexts (MAPREDUCE-1462). It will also
move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message