hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
Date Fri, 28 Oct 2011 12:48:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138267#comment-13138267
] 

Hudson commented on HDFS-2500:
------------------------------

Integrated in Hadoop-Hdfs-0.23-Build #53 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/53/])
    HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes.
Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java

                
> Avoid file system operations in BPOfferService thread while processing deletes
> ------------------------------------------------------------------------------
>
>                 Key: HDFS-2500
>                 URL: https://issues.apache.org/jira/browse/HDFS-2500
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, hdfs-2500.txt
>
>
> While running a workload with concurrent writes and deletes, I saw a lot of NotReplicatedYetExceptions
being thrown due to late arrivals of blockReceived reports from the DN. Looking at the DN
logs, I found that the blockReceived message was being delayed as much as 15 seconds because
the OfferService thread was blocked on file system operations processing deletes. We previously
moved the deletions to another thread, but it still accesses the file system to determine
the block length in the main thread. On a heavily loaded system this can take a long time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message