hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7990) IBR delete ack should not be delayed
Date Sat, 28 Mar 2015 10:41:56 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385230#comment-14385230

Hudson commented on HDFS-7990:

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #146 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/146/])
HDFS-7990. IBR delete ack should not be delayed. Contributed by Daryn Sharp. (kihwal: rev
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

> IBR delete ack should not be delayed
> ------------------------------------
>                 Key: HDFS-7990
>                 URL: https://issues.apache.org/jira/browse/HDFS-7990
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 2.8.0
>         Attachments: HDFS-7990.patch
> HDFS-395 added the incremental BR feature.  A concern was avoiding a race condition with
ack-ing block invalidates followed by the directory scanner re-adding the block before the
async delete service removes the block, possibly resulting in a full BR that includes the
previously delete ack-ed block.
> The solution was to batch & delay block deletion acks via a hardcoded interval of
100 * heartbeat interval (default: 5min).  The delay isn't required:
> #  The {{FSDatasetSpi}} tracks blocks pending deletions precisely so the scanner won't
re-add the block to the volume map
> # Block receiving, received, and deleted go into the same pending report.  A block received
event will trigger an immediate IBR which includes the deletion acks.  Ie. The delay is meaningless
for all but a quiescent cluster
> # Failing to promptly report deleted blocks on a quiescent cluster prevents the NN from
updating the block maps to remove the locations

This message was sent by Atlassian JIRA

View raw message