hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
Date Wed, 10 Sep 2014 22:21:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129242#comment-14129242

Nick Dimiduk commented on HBASE-11331:

Thanks [~enis], [~stack] for having a look.

bq. Add to the simplification of block cache config a note that we need to unify the configuration

Alright, I'll open a ticket.

bq. Needs fat release note and on commit, add something to the refguide in the block cache
section else I'm afraid folks won't come across this feature.

Release note I'd planned, just what's in the commit message; you want more than this?

When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the
BlockCache in their on-disk format. This is different from the default behavior, which decompresses
and decrypts a block before caching.

I'll open a docs ticket for updating the book.

bq. Is this right? Assuming HeapByteBuffer here.

Alas, yes. There's a number of assumptions baked into HFileBlock about HeapByteArrays. The
whole read-ahead of the next block's header feature is based on this supposition (look for
reads beyond limit without checking capacity; this is how I ran into the serialization bug
in the first place).

> [blockcache] lazy block decompression
> -------------------------------------
>                 Key: HBASE-11331
>                 URL: https://issues.apache.org/jira/browse/HBASE-11331
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>             Fix For: 2.0.0, 0.98.7, 0.99.1
>         Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch,
HBASE-11331.03.patch, HBASE-11331.04.patch, HBASE-11331.05-0.98.patch, HBASE-11331.05.patch,
HBASE-11331.06-0.98.patch, HBASE-11331.06.patch, HBASE-11331.07-0.98.patch, HBASE-11331.07.patch,
HBASE-11331LazyBlockDecompressperfcompare.pdf, hbase-hbase-master-hor17n36.gq1.ygridcore.net.log,
lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf,
v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf
> Maintaining data in its compressed form in the block cache will greatly increase our
effective blockcache size and should show a meaning improvement in cache hit rates in well
designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed,
rather than as soon as they're pulled off of disk.
> This is related to but less invasive than HBASE-8894.

This message was sent by Atlassian JIRA

View raw message