hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-877) Client-driven block verification not functioning
Date Thu, 07 Jan 2010 21:49:36 GMT

     [ https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HDFS-877:
-----------------------------

    Description: 
This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out). The issue
is that DFSInputStream relies on readChunk being called one last time at the end of the file
in order to receive the lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
checks pos < getFileLength() before issuing the read. Thus gotEOS never shifts to true
and checksumOk() is never called.



  was:This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out). The
issue is that DFSInputStream relies on readChunk being called one last time at the end of
the file in order to receive the lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
checks pos < getFileLength() before issuing the read. Thus gotEOS never shifts to true
and checksumOk() is never called.

       Priority: Critical  (was: Major)
        Summary: Client-driven block verification not functioning  (was: Client-driven checksum
verification not functioning)

Upgrading this to blocker since it's a cause for a test failure.

Worth noting that this bug only affects the proactive "checksum OK" marking that the client
does after reading an entire block (thus avoiding the periodic scan on the DN). If the checksum
is found to be invalid on the client, it still reports the bad block to the NN just fine.
So, this isn't a dataloss bug, it's just a broken optimization and a failing test.

> Client-driven block verification not functioning
> ------------------------------------------------
>
>                 Key: HDFS-877
>                 URL: https://issues.apache.org/jira/browse/HDFS-877
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>
> This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out). The issue
is that DFSInputStream relies on readChunk being called one last time at the end of the file
in order to receive the lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
checks pos < getFileLength() before issuing the read. Thus gotEOS never shifts to true
and checksumOk() is never called.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message