hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Marc Spaggiari (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8323) Low hanging checksum improvements
Date Mon, 11 Nov 2013 13:06:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818931#comment-13818931

Jean-Marc Spaggiari commented on HBASE-8323:

Will a buffer conversion benefit from the CRC improvement? I mean, if we keept the 'standard'
byte buffers, but convert them to DirectByteBuffer when we call DataCheckSum, will the cost
of this conversion be lower than the benefit of the DataCheckSum? Is so, that might be an
easy change. Else, if we have to chance to DirectByteBuffer all over the place, might be a
bit more difficult...

> Low hanging checksum improvements
> ---------------------------------
>                 Key: HBASE-8323
>                 URL: https://issues.apache.org/jira/browse/HBASE-8323
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance
>            Reporter: Enis Soztutar
> Over at Hadoop land, [~tlipcon] had done some improvements for checksums, a native implementation
for CRC32C (HADOOP-7445) and bulk verify of checksums (HADOOP-7444). 
> In HBase, we can do
>  - Also develop a bulk verify API. Regardless of hbase.hstore.bytes.per.checksum we always
want to verify of the whole checksum for the hfile block.
>  - Enable NativeCrc32 to be used as a checksum algo. It is not clear how much gain we
can expect over pure java CRC32. 
> Though, longer term we should focus on convincing hdfs guys for inline checksums (HDFS-2699)

This message was sent by Atlassian JIRA

View raw message