hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Nykiel" <tom...@fb.com>
Subject Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Date Mon, 11 Jul 2011 21:06:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------

(Updated 2011-07-11 21:06:02.473171)


Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.


Changes
-------

- I removed renaming of the block and meta file from FSDataset. We should consider it in a
separate diff, as suggested by Hairong. By doing so, I need to defer the notification until
the FSAsyncDiskService deletes the files.
- I added an optimization, which does not procuce ACKs for blocks which come from entirely
deleted files (in this situation, the blocks are instantly removed from the blocks map at
the NN side, and hence we need no ACK).


Summary
-------

This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize
the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize
the overhead of processing them, and blocking the namenode for extended time.


This addresses bug HDFS-395.
    https://issues.apache.org/jira/browse/HDFS-395


Diffs (updated)
-----

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1145346

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java
1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1145346

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java
PRE-CREATION 
  trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
1145346 
  trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1145346


Diff: https://reviews.apache.org/r/998/diff


Testing
-------

Will do JUnit tests.


Thanks,

Tomasz


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message