hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-395) DFS Scalability: Incremental block reports
Date Mon, 11 Jul 2011 21:06:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063538#comment-13063538

jiraposter@reviews.apache.org commented on HDFS-395:

This is an automatically generated e-mail. To reply, visit:

(Updated 2011-07-11 21:06:02.473171)

Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.


- I removed renaming of the block and meta file from FSDataset. We should consider it in a
separate diff, as suggested by Hairong. By doing so, I need to defer the notification until
the FSAsyncDiskService deletes the files.
- I added an optimization, which does not procuce ACKs for blocks which come from entirely
deleted files (in this situation, the blocks are instantly removed from the blocks map at
the NN side, and hence we need no ACK).


This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize
the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize
the overhead of processing them, and blocking the namenode for extended time.

This addresses bug HDFS-395.

Diffs (updated)

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1145346

  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1145346 
  trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1145346

  trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1145346

Diff: https://reviews.apache.org/r/998/diff


Will do JUnit tests.



> DFS Scalability: Incremental block reports
> ------------------------------------------
>                 Key: HDFS-395
>                 URL: https://issues.apache.org/jira/browse/HDFS-395
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: blockReportPeriod.patch, explicitDeleteAcks.patch
> I have a cluster that has 1800 datanodes. Each datanode has around 50000 blocks and sends
a block report to the namenode once every hour. This means that the namenode processes a block
report once every 2 seconds. Each block report contains all blocks that the datanode currently
hosts. This makes the namenode compare a huge number of blocks that practically remains the
same between two consecutive reports. This wastes CPU on the namenode.
> The problem becomes worse when the number of datanodes increases.
> One proposal is to make succeeding block reports (after a successful send of a full block
report) be incremental. This will make the namenode process only those blocks that were added/deleted
in the last period.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message