hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Li (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2525) Race between BlockPoolSliceScanner and append
Date Thu, 02 Feb 2012 02:01:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198460#comment-13198460
] 

Brandon Li commented on HDFS-2525:
----------------------------------

This problem is highly reproducible. In my tests, I hit this issue every time with less than
10000 append requests, 100 bytes each time.

In the DataScanner thread, the BlockSender got a block path under rbw. BlockReceiver serving
write request removed the rbw block from rbw directory to finalized directory. The BlockSender
tried to open the file before the BlockReceiver updated the VolumeMap, thus the FileNotFoundException
was thrown.

Here BlockPoolSliceScanner can ignore this exception since a real missing block problem should
be caught by DirectoryScanner.
                
> Race between BlockPoolSliceScanner and append
> ---------------------------------------------
>
>                 Key: HDFS-2525
>                 URL: https://issues.apache.org/jira/browse/HDFS-2525
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Brandon Li
>            Priority: Critical
>
> I wrote a test which runs append() in a loop on a single file with a single replica,
appending 0~100 bytes each time. If this races with the BlockPoolSliceScanner, I observe the
BlockPoolSliceScanner getting FNFE, then reporting the block as bad to the NN. This causes
the writer thread to loop forever on completeFile() since it doesn't see a valid replica.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message