hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
Date Thu, 27 Oct 2011 21:06:32 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Eli Collins updated HDFS-2500:

    Attachment: hdfs-2500-1.patch

Looks great.  +1 pending hudson.

Updated patch attach is rebased on trunk. 
> Avoid file system operations in BPOfferService thread while processing deletes
> ------------------------------------------------------------------------------
>                 Key: HDFS-2500
>                 URL: https://issues.apache.org/jira/browse/HDFS-2500
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2500-1.patch, hdfs-2500.txt
> While running a workload with concurrent writes and deletes, I saw a lot of NotReplicatedYetExceptions
being thrown due to late arrivals of blockReceived reports from the DN. Looking at the DN
logs, I found that the blockReceived message was being delayed as much as 15 seconds because
the OfferService thread was blocked on file system operations processing deletes. We previously
moved the deletions to another thread, but it still accesses the file system to determine
the block length in the main thread. On a heavily loaded system this can take a long time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message