hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6949) Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
Date Mon, 31 Jan 2011 06:49:11 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988664#comment-12988664

Matt Foley commented on HADOOP-6949:

Doug, thanks for bringing up this important use case.  After looking over the code, it appears
there is a single line in WritableRpcEngine where all RPC code calls ObjectWritable.writeObject().
 If this one call is modified to allow the more compact array format, we get the benefit for
all RPC protocols based on WritableRpcEngine, without having to change any of the RPC APIs,
and with no danger to the other callers of ObjectWritable.writeObject().  Any other callers
of ObjectWritable.writeObject() may choose to use the new call, but for safety sake we default
to the old format.

Please take a look at this new patch.  Besides the changes Todd recommended, I broke ObjectWritable.writeObject()
into a stub with the old API signature, and a slightly modified implementation with an additional
boolean argument controlling whether compact format is used for arrays of primitives.  If
false, the behavior is as before; if true the behavior is the new compact format.  Callers
of the old stub always get the old ("false") behavior.  And finally I changed the one line
in WritableRpcEngine to call the new API with the boolean argument "true".

I also re-confirmed that this change gives the improvement in Block Report processing RPC
overhead time.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block
> -------------------------------------------------------------------------------------------------
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>         Attachments: ObjectWritable.diff, arrayprim.patch
>   Original Estimate: 10m
>  Remaining Estimate: 10m
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general
object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message