hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
Date Fri, 03 Jun 2011 17:40:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043904#comment-13043904
] 

Todd Lipcon commented on HDFS-1907:
-----------------------------------

nits:
- positionReadOption is already assigned to false for the first test a few lines above where
you added it
- the new test method shouldn't start with a capital T - our method names are camelCase, not
UpperCamelCase

One question on the logic:
- if the last block is in-progress, but you read from within that block, then readPastEnd
will be false. Hence we will only call getFinalizedBlockRange. But the last block is not finalized.
Does this case work? If so, then that means getFinalizedBlockRange also includes non-finalized
blocks, which is contrary to its name. Here's an example sequence to describe what I mean:
- open file, write one and a half blocks
- call hflush
- another reader asks for the first byte of the second block



> BlockMissingException upon concurrent read and write: reader was doing file position
read while writer is doing write without hflush
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1907
>                 URL: https://issues.apache.org/jira/browse/HDFS-1907
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.0
>         Environment: Run on a real cluster. Using the latest 0.23 build.
>            Reporter: CW Chung
>            Assignee: John George
>         Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, HDFS-1907-5.patch,
HDFS-1907-5.patch, HDFS-1907.patch
>
>
> BlockMissingException is thrown under this test scenario:
> Two different processes doing concurrent file r/w: one read and the other write on the
same file
>   - writer keep doing file write
>   - reader doing position file read from beginning of the file to the visible end of
file, repeatedly
> The reader is basically doing:
>   byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound);
> where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*10000;
> Usually it does not fail right away. I have to read, close file, re-open the same file
a few times to create the problem. I'll pose a test program to repro this problem after I've
cleaned up a bit my current test program.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message