hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
Date Thu, 27 Oct 2011 22:44:32 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HDFS-2500:

    Attachment: hdfs-2500-branch-0.23.txt

branch-23 was slightly different, since AsyncDiskService's bpid parameter had gotten collapsed
into an ExtendedBlock parameter instead of being passed separately. But the differences are
trivial, so I'll commit without a re-review.
> Avoid file system operations in BPOfferService thread while processing deletes
> ------------------------------------------------------------------------------
>                 Key: HDFS-2500
>                 URL: https://issues.apache.org/jira/browse/HDFS-2500
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, hdfs-2500.txt
> While running a workload with concurrent writes and deletes, I saw a lot of NotReplicatedYetExceptions
being thrown due to late arrivals of blockReceived reports from the DN. Looking at the DN
logs, I found that the blockReceived message was being delayed as much as 15 seconds because
the OfferService thread was blocked on file system operations processing deletes. We previously
moved the deletions to another thread, but it still accesses the file system to determine
the block length in the main thread. On a heavily loaded system this can take a long time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message