hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Why datanode does a flush to disk after receiving a packet
Date Thu, 11 Nov 2010 05:11:53 GMT
Nope, flush just flushes the java side buffer to the Linux buffer
cache -- not all the way to the media.

Hsync is the API that will eventually go all the way to disk, but it
has not yet been implemented.


On Wednesday, November 10, 2010, Thanh Do <thanhdo@cs.wisc.edu> wrote:
> Or another way to rephase my question:
> does data.flush and checksumOut.flush guarantee
> data be synchronized with underlying disk,
> just like fsync().
> Thanks
> Thanh
> On Wed, Nov 10, 2010 at 10:26 PM, Thanh Do <thanhdo@cs.wisc.edu> wrote:
>> Hi all,
>> After reading the appenddesign3.pdf in HDFS-256,
>> and looking at the BlockReceiver.java code in 0.21.0,
>> I am confused by the following.
>> The document says that:
>> *For each packet, a DataNode in the pipeline has to do 3 things.
>> 1. Stream data
>>       a. Receive data from the upstream DataNode or the client
>>       b. Push the data to the downstream DataNode if there is any
>> 2. Write the data/crc to its block file/meta file.
>> 3. Stream ack
>>       a. Receive an ack from the downstream DataNode if there is any
>>       b. Send an ack to the upstream DataNode or the client*
>> And *"...there is no guarantee on the order of (2) and (3)"*
>> In BlockReceiver.receivePacket(), after read the packet buffer,
>> datanode does:
>> 1) put the packet seqno in the ack queue
>> 2) write data and checksum to disk
>> 3) flush data and checksum (to disk)
>> The thing that confusing me is that: the streaming of ack does not
>> necessary depends on whether data has been flush to disk or not.
>> Then, my question is:
>> Why do DataNode need to flush data and checksum
>> every time the DataNode receives a packet. This flush may be costly.
>> Why cant the DataNode just batch server write (after receiving
>> server packet) and flush all at once?
>> Is there any particular reason for doing so?
>> Can somebody clarify this for me?
>> Thanks so much.
>> Thanh

Todd Lipcon
Software Engineer, Cloudera

View raw message