Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 6 Aug 2014 19:36:12 +0000 (UTC)
From: "Nick Dimiduk (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12720693.1402520063518.28013.1407353772595@arcas>
In-Reply-To: <JIRA.12720693.1402520063518@arcas>
References: <JIRA.12720693.1402520063518@arcas>
Subject: [jira] [Updated] (HBASE-11331) [blockcache] lazy block
 decompression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick Dimiduk updated HBASE-11331:
---------------------------------

    Attachment: lazy-decompress.02.1.pdf

re: lazy-decompress.02.1.pdf

Attaching some metrics after running v02 with less os page cache and feature enabled/disabled.

This time the total RS heap is increased to 20g, so the effective blockcache size is ~12g. The machine has 24g of RAM, so this squeezes the OS-available memory down to ~4g -- when the blockcache is full. I've added blockcache size and OS page cache to the charts  Same table, same compression ratio, same PE randomRead test, this time run with --size=45, so I expect ~11g of compressed blocks.

I expected this to be an IO-bound workload with flag=false but to be cache-at-capacity for flag=true. The massive evictions look to me like the blockcache heapsize calculation is based on the decompressed size, not compressed size. Updating patch.

> [blockcache] lazy block decompression
> -------------------------------------
>
>                 Key: HBASE-11331
>                 URL: https://issues.apache.org/jira/browse/HBASE-11331
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>         Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.pdf
>
>
> Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk.
> This is related to but less invasive than HBASE-8894.


--
This message was sent by Atlassian JIRA
(v6.2#6252)