Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of doducthanhbk@gmail.com
 designates 209.85.214.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=UmiFst6OthKOVdHrCh8HAXdu9tpH4HGDQFbEEBLMGNn+Rjlzu9+ohgjwtLo1Zsbmw6
         92DAJyJYN1ZxBl+XdZSdog9oggKU9X1jhu2z/CdhZ1OLjKPP7od7v9kyqBIDe3q1Wn75
         6HVUXtBEHXbvcWgWWMjc5qmGGsIEUHaQCZuUg=
MIME-Version: 1.0
Sender: doducthanhbk@gmail.com
Date: Wed, 10 Nov 2010 22:26:15 -0600
Message-ID: <AANLkTinQbvOzH76_KcVZDMh4HqYH+t2dB7Q7etyHMqXR@mail.gmail.com>
Subject: Why datanode does a flush to disk after receiving a packet
From: Thanh Do <thanhdo@cs.wisc.edu>
To: hdfs-dev@hadoop.apache.org, hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646dc3ceefb0d0494bf5fea

--00163646dc3ceefb0d0494bf5fea
Content-Type: text/plain; charset=UTF-8

Hi all,

After reading the appenddesign3.pdf in HDFS-256,
and looking at the BlockReceiver.java code in 0.21.0,
I am confused by the following.

The document says that:
*For each packet, a DataNode in the pipeline has to do 3 things.
1. Stream data
      a. Receive data from the upstream DataNode or the client
      b. Push the data to the downstream DataNode if there is any
2. Write the data/crc to its block file/meta file.
3. Stream ack
      a. Receive an ack from the downstream DataNode if there is any
      b. Send an ack to the upstream DataNode or the client*

And *"...there is no guarantee on the order of (2) and (3)"*

In BlockReceiver.receivePacket(), after read the packet buffer,
datanode does:
1) put the packet seqno in the ack queue
2) write data and checksum to disk
3) flush data and checksum (to disk)

The thing that confusing me is that: the streaming of ack does not
necessary depends on whether data has been flush to disk or not.
Then, my question is:
Why do DataNode need to flush data and checksum
every time the DataNode receives a packet. This flush may be costly.
Why cant the DataNode just batch server write (after receiving
server packet) and flush all at once?
Is there any particular reason for doing so?

Can somebody clarify this for me?

Thanks so much.
Thanh

--00163646dc3ceefb0d0494bf5fea
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi all, <br><br>After reading the appenddesign3.pdf in HDFS-256,<br>and loo=
king at the BlockReceiver.java code in 0.21.0,<br>I am confused by the foll=
owing.<br><br>The document says that:<br><i>For each packet, a DataNode in =
the pipeline has to do 3 things.<br>
1. Stream data<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a. Receive data from the u=
pstream DataNode or the client<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 b. Push th=
e data to the downstream DataNode if there is any<br>2. Write the data/crc =
to its block file/meta file.<br>3. Stream ack<br>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a. Receive an ack from the downstream DataNo=
de if there is any<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 b. Send an ack to the =
upstream DataNode or the client</i><br><br>And <i>&quot;...there is no guar=
antee on the order of (2) and (3)&quot;</i><br>
<br>In BlockReceiver.receivePacket(), after read the packet buffer,<br>data=
node does:<br>1) put the packet seqno in the ack queue<br>2) write data and=
 checksum to disk<br>3) flush data and checksum (to disk)<br><br>The thing =
that confusing me is that: the streaming of ack does not <br>
necessary depends on whether data has been flush to disk or not.<br>Then, m=
y question is: <br>Why do DataNode need to flush data and checksum<br>every=
 time the DataNode receives a packet. This flush may be costly.<br>Why cant=
 the DataNode just batch server write (after receiving <br>
server packet) and flush all at once?<br>Is there any particular reason for=
 doing so?<br><br>Can somebody clarify this for me?<br><br>Thanks so much.<=
br>Thanh<br><br><br><br>

--00163646dc3ceefb0d0494bf5fea--