hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6949) Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
Date Fri, 25 Feb 2011 07:30:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999245#comment-12999245
] 

Konstantin Shvachko commented on HADOOP-6949:
---------------------------------------------

- I recommend looking for subclasses of VersionedProtocol rather than search. You can see
that most of the protocols you listed implement VersionedProtocol, except for two AvroRpcEngine
and WritableRpcEngine, which implement RpcEngine. Avro has its own serialization object, afaiu,
so we don't need to change version for the latter two.
- I think it is fine to change only ipc.Server#CURRENT_VERSION, as Hairong sugested, which
is the VersionedProtocol version, rather than all subclasses, as it will provide incompatibility
for all of them at once.
- As I said in my [comment|https://issues.apache.org/jira/browse/HDFS-1583?focusedCommentId=12981404&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12981404]
in HDFS-1583, Java does not support heterogeneous arrays, see [ArrayStoreException|http://download.oracle.com/javase/6/docs/api/java/lang/ArrayStoreException.html]
for the reference. So we don't need to support such generic semantic. We can assume the array
is strictly homogeneous.
- Yes we need to support only arrays of Writables (and of primitive types). 
- NULL values should be supported.
- My opinion this is important optimization to be included in 0.22. If people have doubts
I can start a vote on general. Lmk.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block
reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general
object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message