hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16454) Compactor's shipping logic decision should be based on the current store's block size
Date Fri, 19 Aug 2016 11:42:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428037#comment-15428037
] 

ramkrishna.s.vasudevan commented on HBASE-16454:
------------------------------------------------

bq.I mean whether this minFilesToCompact is the number of files getting compacted in this
particular compaction. Or is it the min one (def to 3)
It is the one that is configured
{code}
    long minFilesToCompact = Math.max(2L,
        conf.getInt(CompactionConfiguration.HBASE_HSTORE_COMPACTION_MIN_KEY,
            /* old name */ conf.getInt("hbase.hstore.compactionThreshold", 3)));
{code}
So sometimes if this value is also set to some higher value we may end up in much bigger numbers.


> Compactor's shipping logic decision should be based on the current store's block size
> -------------------------------------------------------------------------------------
>
>                 Key: HBASE-16454
>                 URL: https://issues.apache.org/jira/browse/HBASE-16454
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> Currently we do call KeyValueScanner.shipped() if the size of the KVs compacted reaches
certain threshold
> {code}
>      if (kvs != null && bytesWrittenProgressForShippedCall > shippedCallSizeLimit)
{
>             kvs.shipped();
>             bytesWrittenProgressForShippedCall = 0;
> }
> {code}
> Where the shippedCallSizeLimit is
> {code}
>     long shippedCallSizeLimit = (long) minFilesToCompact * HConstants.DEFAULT_BLOCKSIZE;
> {code}
> Instead it is better if it is based on the actual configured block size for that store.
It can still have the multiplying factore of 'minFilesToCompact ' but having default store
size would mean that if the store has smaller block sizes configured we hold on to more blocks
and those will not get evicted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message