Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of zhengda1936@gmail.com
 designates 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=cNGcMW8vnGwsyIa8x7ihWm3kkGxv6fFbj4TslmQe/Q+zKagqcsiVrzSpxf9YElRU7O
         H6ViNp05FlxBKlzpCaVHWdg3+gKWgOqGdQs16gFruZb6IDtClgH4AZOHxCIQaZ7/12QX
         pezCh7fcMR4xtipN9iL5a71Era9mkrIXA/6wQ=
Message-ID: <4D4384C1.5060709@gmail.com>
Date: Fri, 28 Jan 2011 22:08:49 -0500
From: Da Zheng <zhengda1936@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
 rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: message transmission in Hadoop
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello,

I monitored system calls of HDFS with systemtap and found HDFS actually sends
many 1-byte data to the network. I could also see many 8-byte and 64-byte data
written to the OS though I don't know whether they are written to the disk or
sent to the network. I did see many 8-byte data sent to the network. The number
of these data is several times more than 64KB data packet sent by HDFS.

Could anyone tell me why HDFS sends so many small packets? heartbeat messages?
RPCs? It doesn't seem to me these messages can be just 1 byte.

Thanks,
Da