hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10291) TestShortCircuitLocalRead failing
Date Thu, 14 Apr 2016 12:45:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241066#comment-15241066

Steve Loughran commented on HDFS-10291:

1. test is doing a short circuited read of a 13 byte file {{doTestShortCircuitRead(false,
size= 13, readOffset= 0)}}
2. # creates 13 bytes of data, saves it: {{byte[] fileData = AppendTestUtil.randomBytes(seed,
3. dest buffer is created for this
byte[] actual = new byte[expected.length-readOffset];
4 does some small reads:
//Read a small number of bytes first.
int nread = stm.read(actual, 0, 3);
nread += stm.read(actual, nread, 2);
//Read across chunk boundary
nread += stm.read(actual, nread, 517);    // ** HERE **

The exception is being raised as the code is asking to read 517 bytes into a buffer 13 bytes
long. This breaks IOStream's rules: you can't ask for more than you have space for. It says
that clearly in the IOStream API spec; what was added in HADOOP-12994 was the checking of
passing in too big a length or negative offsets.

I think this a bug in the test. Whatever it is trying to do, it shouldn't be trying to do
it on such a small buffer.

What's interesting though is when you delve into the code: the block reader logic doesn't
look at the length of the read at all. That is, it appears to fill up the entire byte array
passed in, from the offset supplied, stopping at the end of the buffer or file, whichever
comes first.

Which is something that other code (i.e. production code) could be relying on. They shouldn't,
as the code will break when working with any FS other than HDFS, but there is a risk that
they might.

What to do?

# retain checks, fix test.
# log at warning and shrink len parameter when passed down. People shouldn't be doing this,
but HDFS will reluctantly let you.

> TestShortCircuitLocalRead failing
> ---------------------------------
>                 Key: HDFS-10291
>                 URL: https://issues.apache.org/jira/browse/HDFS-10291
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> {{TestShortCircuitLocalRead}} failing as length of read is considered off end of buffer.
There's an off-by-one error somewhere in the test or the new validation code

This message was sent by Atlassian JIRA

View raw message