Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative;
	boundary="===============3523079121054304053=="
MIME-Version: 1.0
Subject: Re: Review Request: HDFS-395 DFS Scalability: Incremental block
 reports
From: "Tomasz Nykiel" <tomasz@fb.com>
To: "Dhruba Borthakur" <dhruba@gmail.com>, "Hairong Kuang" <hairong@fb.com>
Date: Mon, 11 Jul 2011 21:06:02 -0000
Message-ID: <20110711210602.21485.39092@reviews.apache.org>
Cc: "Tomasz Nykiel" <tomasz@fb.com>,
 "hadoop-hdfs" <hdfs-dev@hadoop.apache.org>
In-Reply-To: <20110701203417.29522.30677@reviews.apache.org>
References: <20110701203417.29522.30677@reviews.apache.org>

--===============3523079121054304053==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------

(Updated 2011-07-11 21:06:02.473171)


Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.


Changes
-------

- I removed renaming of the block and meta file from FSDataset. We should c=
onsider it in a separate diff, as suggested by Hairong. By doing so, I need=
 to defer the notification until the FSAsyncDiskService deletes the files.
- I added an optimization, which does not procuce ACKs for blocks which com=
e from entirely deleted files (in this situation, the blocks are instantly =
removed from the blocks map at the NN side, and hence we need no ACK).


Summary
-------

This patch introduces explicit ACKs sent form datanodes to the namenode in =
order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which i=
n turn will minimize the overhead of processing them, and blocking the name=
node for extended time.


This addresses bug HDFS-395.
    https://issues.apache.org/jira/browse/HDFS-395


Diffs (updated)
-----

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockMa=
nager.java 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java =
1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java=
 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsync=
DiskService.java 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.j=
ava 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java =
1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.j=
ava 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtoc=
ol.java 1145346 =

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDelete=
dBlockInfo.java PRE-CREATION =

  trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThrough=
putBenchmark.java 1145346 =

  trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadD=
atanode.java 1145346 =


Diff: https://reviews.apache.org/r/998/diff


Testing
-------

Will do JUnit tests.


Thanks,

Tomasz


--===============3523079121054304053==--