hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
Date Thu, 02 Jun 2011 22:23:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043103#comment-13043103

Daryn Sharp commented on HDFS-1907:

I just did a cursory read of {{getFinalizedBlockRange}}.  If the file is committed, it appears
to throw an exception if the client attempts to read past the end of file.  However, in the
case where the file has an uncompleted block, I think there's a secondary bug where no exception
is thrown if the client is attempting to read past the end of the uncompleted block.  Basically
a final bounds check should be added.

Small issue with the test.  I know you probably copied-n-pasted, however {{assertTrue(stat
== 0)}} is an anti-pattern.  {{assertTrue}} should used to test a real boolean value, not
a fabricated boolean within the assert.  This condition needs to be {{assertEquals(0, stat)}}.
 The reason is if {{assertTrue}} fails, junit can't report the bad value.  Whereas {{assertEquals}}
will report "I expected *this*, but got *that*".

If you feel esp. industrious, you might pay it forward and use a regexp to search and replace
the other similar conditions to {{assertEquals}}...  The next dev that breaks a test will
thank you.  How much you do is up to you, but at least fix the new test.

> BlockMissingException upon concurrent read and write: reader was doing file position
read while writer is doing write without hflush
> ------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HDFS-1907
>                 URL: https://issues.apache.org/jira/browse/HDFS-1907
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.0
>         Environment: Run on a real cluster. Using the latest 0.23 build.
>            Reporter: CW Chung
>            Assignee: John George
>         Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, HDFS-1907.patch
> BlockMissingException is thrown under this test scenario:
> Two different processes doing concurrent file r/w: one read and the other write on the
same file
>   - writer keep doing file write
>   - reader doing position file read from beginning of the file to the visible end of
file, repeatedly
> The reader is basically doing:
>   byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound);
> where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*10000;
> Usually it does not fail right away. I have to read, close file, re-open the same file
a few times to create the problem. I'll pose a test program to repro this problem after I've
cleaned up a bit my current test program.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message