Return-Path: 
 <hdfs-issues-return-150164-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDEE818E5F
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Tue, 19 Jan 2016 12:47:40 +0000 (UTC)
Received: (qmail 93011 invoked by uid 500); 19 Jan 2016 12:47:40 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 92908 invoked by uid 500); 19 Jan 2016 12:47:40 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 92774 invoked by uid 99); 19 Jan 2016 12:47:39 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2016 12:47:39 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id C22B62C1F55
	for <hdfs-issues@hadoop.apache.org>; Tue, 19 Jan 2016 12:47:39 +0000 (UTC)
Date: Tue, 19 Jan 2016 12:47:39 +0000 (UTC)
From: "Kai Zheng (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12932403.1453207644000.147252.1453207659792@Atlassian.JIRA>
In-Reply-To: <JIRA.12932403.1453207644000@Atlassian.JIRA>
References: <JIRA.12932403.1453207644000@Atlassian.JIRA>
 <JIRA.12932403.1453207644413@arcas>
Subject: [jira] [Created] (HDFS-9663) Optimize some RPC call using lighter
 weight construct than DatanodeInfo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Kai Zheng created HDFS-9663:
-------------------------------

             Summary: Optimize some RPC call using lighter weight construct than DatanodeInfo
                 Key: HDFS-9663
                 URL: https://issues.apache.org/jira/browse/HDFS-9663
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Kai Zheng
            Assignee: Kai Zheng


While working on HDFS-8430 when add a RPC in DataTransferProtocol, it was noticed the very heavy construct either {{DatanodeInfo}} or {{DatanodeInfoWithStorage}} is used to represent a datanode just for connection in most time. However, it's very fat and contains much more information than that needed. See how it's defined:
{code}
public class DatanodeInfo extends DatanodeID implements Node {
  private long capacity;
  private long dfsUsed;
  private long remaining;
  private long blockPoolUsed;
  private long cacheCapacity;
  private long cacheUsed;
  private long lastUpdate;
  private long lastUpdateMonotonic;
  private int xceiverCount;
  private String location = NetworkTopology.DEFAULT_RACK;
  private String softwareVersion;
  private List<String> dependentHostNames = new LinkedList<>();
  private String upgradeDomain;
...
{code}
In client and datanode sides, for RPC calls like {{DataTransferProtocol#writeBlock}}, looks like the information contained in {{DatanodeID}} is almost enough.

I did a quick hack that using a light weight construct like {{SimpleDatanodeInfo}} that simply extends DatanodeID (no other field added, but if whatever field needed, then just add it) and changed the DataTransferProtocol#writeBlock call. Manually checked many relevant tests it did work fine. How much network traffic saved, did a simple test with codes in {{Sender}}:
{code}
  private static void send(final DataOutputStream out, final Op opcode,
      final Message proto) throws IOException {
    LOG.trace("Sending DataTransferOp {}: {}",
        proto.getClass().getSimpleName(), proto);
    int before = out.size();
    op(out, opcode);
    proto.writeDelimitedTo(out);
    int after = out.size();
    System.out.println("XXXXXXXXXXXXXXXXX sent=" + (after - before));
    out.flush();
  }
{code}
Ran the test {{TestWriteRead#testWriteAndRead}}, the change can  save about 100 bytes in most time for the call. The saving may be not so big because only 3 datanodes are to send, but in situations like in {{BlockECRecoveryCommand}}, there can be 6+ 3 datanodes as targets and sources to send, the saving will be significant.

Hence, suggest use more light weight construct to represent a datanode in RPC calls when possible. Or other ideas to avoid unnecessary wire data size. This may make sense, as noted, there were some discussions in HDFS-8999 to save some datanodes bandwidth.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)