hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allan Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
Date Thu, 09 Mar 2017 11:32:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902914#comment-15902914
] 

Allan Yang edited comment on HBASE-17757 at 3/9/17 11:31 AM:
-------------------------------------------------------------

{quote}
Why would we not just enable this behavior always? What feedback do i get that tells me I've
chosen a bad UNIFY_ENCODED_BLOCKSIZE_RATIO ? Thank you.
{quote}
For long time, blockszie is stand for block size before encoding, user may have carefully
chose a blocksize to balance the performance. if we enable this feature always, after upgrading
their HBase to a new version with this patch, the actual blocksize will be changed. It may
not be friendly to some users. So in our company, we made a switch, only deploying this feature
for new customers. Old customers who want to use this feature need to carefully choose a UNIFY_ENCODED_BLOCKSIZE_RATIO
.
{quote}
What feedback do i get that tells me I've chosen a bad UNIFY_ENCODED_BLOCKSIZE_RATIO ?
{quote}
No feedback, I'm afraid. Like blocksize, HBase users is responsible for their choose.  A bad
'blocksize' or ' UNIFY_ENCODED_BLOCKSIZE_RATIO' may compromise performance, but no other damage.

{quote}
 Can we just reduce the #configs here? Say by default we want this feature to be off then
give this ratio as 1 only?
{quote}
[~anoopsamjohn], some problem here, blockszie is stand for block size before encoding, if
we use only ratio to control this feature, blocksize will become block size after encoding

After all, If we add a new config for hfile block, for example, 'encodedBlocksize', if it
is set, it will override 'blocksize', then we can introduce only one new config. What do you
think? [~stack], [~anoopsamjohn]



was (Author: allan163):
{quote}
Why would we not just enable this behavior always? What feedback do i get that tells me I've
chosen a bad UNIFY_ENCODED_BLOCKSIZE_RATIO ? Thank you.
{quote}
For long time, blockszie is stand for block size before encoding, user may have carefully
chose a blocksize to balance the performance. if we enable this feature always, after upgrading
their HBase to a new version with this patch, the actual blocksize will be changed. It may
not be friendly to some users. So in our company, we made a switch, only deploying this feature
for new customers. Old customers who want to use this feature need to carefully choose a UNIFY_ENCODED_BLOCKSIZE_RATIO
.
{quote}
What feedback do i get that tells me I've chosen a bad UNIFY_ENCODED_BLOCKSIZE_RATIO ?
{quote}
No feedback, I'm afraid. Like blocksize, HBase users is responsible for their choose.  A bad
'blocksize' or ' UNIFY_ENCODED_BLOCKSIZE_RATIO' may compromise performance, but no other damage.

{quote}
 Can we just reduce the #configs here? Say by default we want this feature to be off then
give this ratio as 1 only?
{quote}
[~anoopsamjohn], some problem here, blockszie is stand for block size before encoding, if
we use only ratio to control this feature, blocksize will become block size after encoding

After all, If we add a new config for hfile block, for example, 'encodedBlocksize', if it
is set, it will override 'blocksize', then we can introduce only one new config.


> Unify blocksize after encoding to decrease memory fragment 
> -----------------------------------------------------------
>
>                 Key: HBASE-17757
>                 URL: https://issues.apache.org/jira/browse/HBASE-17757
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Allan Yang
>            Assignee: Allan Yang
>         Attachments: HBASE-17757.patch
>
>
> Usually, we store encoded block(uncompressed) in blockcache/bucketCache. Though we have
set the blocksize, after encoding, blocksize is varied. Varied blocksize will cause memory
fragment problem, which will result in more FGC finally.In order to relief the memory fragment,
This issue adjusts the encoded block to a unified size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message