hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-877) Client-driven block verification not functioning
Date Fri, 08 Jan 2010 02:36:13 GMT

     [ https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HDFS-877:

    Attachment: hdfs-877.txt

I took the following route to fixing this:
- BlockReader now knows how many bytes it's expected to read.
- After it has read this many bytes, it reads ahead one packet, expecting an empty "end of
stream" packet from the DN. In the case that this doesn't come (or it's not the special end
of stream packet) it throws IOE. If it does get it, it sets gotEOS.

This current patch is a little ugly since it duplicates the packet-header-reading logic. I
have another version that depends on HDFS-881 to clean that up, if 881 gets committed in the

> Client-driven block verification not functioning
> ------------------------------------------------
>                 Key: HDFS-877
>                 URL: https://issues.apache.org/jira/browse/HDFS-877
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client, test
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hdfs-877.txt
> This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out). The issue
is that DFSInputStream relies on readChunk being called one last time at the end of the file
in order to receive the lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
checks pos < getFileLength() before issuing the read. Thus gotEOS never shifts to true
and checksumOk() is never called.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message