hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LiuLei (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
Date Sun, 23 Sep 2012 05:09:09 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461321#comment-13461321

LiuLei commented on HBASE-6868:

Hi all,

1. Local Read
When we set dfs.client.read.shortcircuit=true and dfs.client.read.shortcircuit.skip.checksum=false,
and verifyChecksum parameter is ture in BlockReaderLocal constructor,   the DFSClient read
meta file and verify checksum.

When we set dfs.client.read.shortcircuit=true and dfs.client.read.shortcircuit.skip.checksum=true,
the DFSClient don't read meta file and don't verify checksum.

2. Remote Read
When we call DistributedFileSystem.setVerifyChecksum(false),  the DFSClient don't  verify
When we call DistributedFileSystem.setVerifyChecksum(true),  the DFSClient  verify checksum.

the verifyChecksum property default value is true.

I think  there is another problem in local read,   BlockReaderLocal class use  "static Map<Integer,
LocalDatanodeInfo> localDatanodeInfoMap" property to store local block file path and local
meta file path. When I stop HDFS cluster or I kill the local DataNode and delete file use
"./hadoop dfs -rm path" command ,  the RegionServer still can read the data form local file.
I think that may lead to RegionServer read wrong data.

> Skip checksum is broke; are we double-checksumming by default?
> --------------------------------------------------------------
>                 Key: HBASE-6868
>                 URL: https://issues.apache.org/jira/browse/HBASE-6868
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile, wal
>    Affects Versions: 0.94.0, 0.94.1
>            Reporter: LiuLei
>            Priority: Blocker
>             Fix For: 0.94.3, 0.96.0
> The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't
need to read the checksum from meta file of HDFS.  But HLog file of Hbase don't contain the
checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS.  We
could  add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip
checksum facility is enabled 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message