hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Liochon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8323) Low hanging checksum improvements
Date Wed, 06 Nov 2013 09:29:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814737#comment-13814737
] 

Nicolas Liochon commented on HBASE-8323:
----------------------------------------

I saw it while looking at a server. The class NativeCrc32 is not public in hadoop common,
(while PureJavaCrc32C is public, in the java & hadoop meanings of the terms). The easiest
way for HBase would be to reuse directly NativeCrc32...  [~todd@lipcon.org], is there any
problem if NativeCrc32 becomes public, or at least LimitedPrivate?

> Low hanging checksum improvements
> ---------------------------------
>
>                 Key: HBASE-8323
>                 URL: https://issues.apache.org/jira/browse/HBASE-8323
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance
>            Reporter: Enis Soztutar
>
> Over at Hadoop land, [~tlipcon] had done some improvements for checksums, a native implementation
for CRC32C (HADOOP-7445) and bulk verify of checksums (HADOOP-7444). 
> In HBase, we can do
>  - Also develop a bulk verify API. Regardless of hbase.hstore.bytes.per.checksum we always
want to verify of the whole checksum for the hfile block.
>  - Enable NativeCrc32 to be used as a checksum algo. It is not clear how much gain we
can expect over pure java CRC32. 
> Though, longer term we should focus on convincing hdfs guys for inline checksums (HDFS-2699)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message