hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6949) Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
Date Fri, 10 Sep 2010 14:30:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908036#action_12908036

Todd Lipcon commented on HADOOP-6949:

Hi Navis,

Please format your change as a "patch" file that can be applied against the trunk of the hadoop-common
repository. This one has a diff of 1.txt and 2.txt which makes it more difficult to apply.

Regarding the change, it seems good except that it's backwards-incompatible for ObjectWritable.
Since ObjectWritable is a public facing class, it's likely that some people have stored them
on disk (eg in sequencefiles) so I don't think we can make a breaking change like this.

Do you have benchmarks that show a significant improvement by making this change? If so, it
may be worth creating a new CompactObjectWritable that's used only in the RPC layer, for example,
and marked as a Private Evolving interface. If the improvement is not significant, we should
skip this change since we'll eventually move to a more compact RPC format (Avro) anyway.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block
> -------------------------------------------------------------------------------------------------
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Priority: Trivial
>         Attachments: ObjectWritable.diff
>   Original Estimate: 0.17h
>  Remaining Estimate: 0.17h
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general
object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message