Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2094ED1E9 for ; Wed, 26 Dec 2012 23:44:17 +0000 (UTC) Received: (qmail 19591 invoked by uid 500); 26 Dec 2012 23:44:15 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 19547 invoked by uid 500); 26 Dec 2012 23:44:15 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 19472 invoked by uid 99); 26 Dec 2012 23:44:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Dec 2012 23:44:15 +0000 Date: Wed, 26 Dec 2012 23:44:15 +0000 (UTC) From: "Sanjay Radia (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-9163) The rpc msg in ProtobufRpcEngine.proto should be moved out to avoid an extra copy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539751#comment-13539751 ] Sanjay Radia commented on HADOOP-9163: -------------------------------------- Todd, I was planning to change Hadoop rpc to use CodedOutputStream to reduce copies; this requires no changes in protocol and is merely code change in how messages are serialized. However, the nested request field will cause an extra copy. See HBases's ProtobufRpcCLientEngine#constructRpcRequest {code} RpcRequestBody constructRpcRequest(Method method, Object[] params) throws ServiceException { RpcRequestBody.Builder builder = RpcRequestBody.newBuilder(); ... param = (Message)params[0]; ... builder.setRequest(param.toByteString()); ... } {code} The param.toByteString will cause the rpc request message to be serialized as bytes. This will be serialized once again when RpcRequestBody is serialized later. (There is similar code in Hadoop RPC since HBase rpc was copied from Hadoop rpc as far as I understand.) Am i mistaken here? I have tried to see if could have fixed this in a compatible way by setting the type of Request to be "message" instead of "bytes" as this would not change the protocol since bytes and message are the same; however I cannot define a field to be "message" in a .proto file. Hence my suggesting to move it out. I have not marked this as jira as incompatible yet since I am still exploring a couple of options. > The rpc msg in ProtobufRpcEngine.proto should be moved out to avoid an extra copy > ---------------------------------------------------------------------------------- > > Key: HADOOP-9163 > URL: https://issues.apache.org/jira/browse/HADOOP-9163 > Project: Hadoop Common > Issue Type: Sub-task > Reporter: Sanjay Radia > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira