hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
Date Tue, 25 Oct 2011 17:44:32 GMT
Avoid file system operations in BPOfferService thread while processing deletes
------------------------------------------------------------------------------

                 Key: HDFS-2500
                 URL: https://issues.apache.org/jira/browse/HDFS-2500
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: data-node
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
         Attachments: hdfs-2500.txt

While running a workload with concurrent writes and deletes, I saw a lot of NotReplicatedYetExceptions
being thrown due to late arrivals of blockReceived reports from the DN. Looking at the DN
logs, I found that the blockReceived message was being delayed as much as 15 seconds because
the OfferService thread was blocked on file system operations processing deletes. We previously
moved the deletions to another thread, but it still accesses the file system to determine
the block length in the main thread. On a heavily loaded system this can take a long time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message